Publish a self-contained Nemotron 3 Super Q4 + MTPv2 + suffix N-gram layer package

[i386]

Goal

Build and publish a self-contained Skippy layer package combining:

• the existing Nemotron 3 Super UD-Q4_K_XL target;
• NVIDIA's separately released BF16 MTPv2 head;
• Daniel Winter-Wijntjes' request-local exact-suffix N-gram proposer; and
• package-owned native-MTP, suffix N-gram, and composite MTP+suffix generation strategies.

The intended product strategy is mtp-suffix: native MTP predicts the uncertain prefix, suffix N-gram lookup extends it from earlier request-local prompt/tool history, and the target verifies the combined candidate. The package should download, plan, split, and serve like any other Mesh layer package, without an external draft_model_path and without requantizing the complete 120B target.

Background

The current target is:

• source GGUF: unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF;
• quant: UD-Q4_K_XL;
• Mesh package: meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-layers;
• target size: approximately 84.49 GB; and
• trunk layers: 88.

Inspection of the published GGUF shards confirms that they contain only blk.0 through blk.87: there are no MTP/NextN tensors or num_nextn_predict_layers metadata. The current package consequently reports layer_count: 88 and generation: null.

NVIDIA now publishes an updated MTP head separately:

• nvidia/Nemotron-3-Super-120B-A12B-BF16-MTPv2;
• approximately 3B parameters / 5.88 GB BF16;
• num_nextn_predict_layers = 1; and
• intended to be attached to a compatible Nemotron 3 Super target rather than served as a standalone model.

NVIDIA reports mean accepted-token lengths at draft length 7 of 4.86 for coding and 4.31 overall, compared with 3.78 and 3.45 for the original built-in MTP head. These are acceptance lengths, not direct end-to-end speedup figures, and must be remeasured on the Q4 target and Apple Silicon lab.

Why MTP plus suffix N-gram

The suffix N-gram proposer contributed in #1037 is aimed directly at coding-agent and tool-loop traffic:

• file re-emission with localized edits;
• repeated structured configuration;
• repeated identifiers;
• echoed tool output; and
• long verbatim prompt regions.

Native MTP and suffix retrieval have complementary roles:

target-committed history ──► request-local exact-suffix index
          │                              │
          ├──► native MTP prefix ────────┤ read-only lookup query
          │                              ▼
          │                       copied suffix tail
          │                              │
          └──────────────► [MTP prefix | suffix tail]
                                      │
                                      ▼
                              target verification

The MTP continuation is read-only lookup context and never enters suffix history until the target commits it. History remains request-local and is never shared between users or sessions. The target remains authoritative for every token.

The package must retain separate mtp and ngram-suffix strategies as controls so the composite result remains attributable.

Why a layer package

A self-contained layer package provides:

• one model reference for discovery, download, and serving;
• no machine-local MTP sidecar configuration;
• package-owned selection of the composite strategy;
• automatic native-MTP discovery through model-package.json;
• topology and memory planning against the complete artifact set;
• placement of the MTP weights only on the final/output stage;
• reuse of the target's existing Q4 embedding, final norm, and LM head;
• atomic versioning of the target/MTP pairing; and
• normal package preflight, materialization, and cache behavior.

Publish this as a new distribution rather than replacing the existing non-MTP package.

Suggested distribution id:

NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-layers

Existing runtime support

The required generic runtime pieces already exist:

• pinned llama.cpp supports Nemotron-H MTP conversion and execution;
• the Nemotron-H loader supports an appended NextN/MTP block;
• Skippy exposes native MTP proposal and target verification;
• ngram-suffix provides a request-local exact-seed/suffix index with up to a 64-token match horizon;
• the speculative frontend supports native-mtp+ngram-suffix composition;
• the package schema supports native-mtp, ngram-suffix, and composite strategies; and
• package preflight validates proposer types, request-local history, MTP layer indices, and extension policy.

This issue is a composition, package-generation, publication, and certification task. It should not introduce another speculative-decoding implementation.

Package shape

The combined model should contain:

• 88 target trunk blocks: blk.0 through blk.87;
• one appended MTP block: blk.88;
• nextn_predict_layers = 1;
• the existing Q4 target embeddings, final norm, and LM head; and
• BF16 MTPv2 tensors initially.

Expected size:

• existing target: approximately 84.49 GB;
• BF16 MTPv2 addition: approximately 5.88 GB; and
• combined package: approximately 90.37 GB before minor packaging overhead.

The package should report 89 logical blocks: 88 target blocks plus the appended MTP block. Suffix N-gram state is request-local runtime state and adds no model-weight artifact.

Composite GGUF construction

Create a reproducible composition workflow that:

1. Reads all shards of the existing UD-Q4_K_XL GGUF.
2. Copies every target tensor byte-for-byte, preserving its existing quantization.
3. Converts NVIDIA's MTPv2 SafeTensors into GGUF tensor form.
4. Appends the MTP tensors as blk.88.
5. Updates the combined GGUF metadata:
   • block count: 89;
   • target trunk count: 88;
   • NextN prediction layers: 1; and
   • per-layer attention/feed-forward metadata covering blk.88.
6. Preserves the target tokenizer and chat-template metadata.
7. Generates the Skippy layer package from the composite model.

The complete 120B BF16 target must not be downloaded, converted, or requantized. Reuse the Q4 target's existing shared tensors, avoiding the approximately 2.15 GB of duplicated BF16 embedding/output tensors required by a standalone MTP sidecar.

Keep GGUF tensor inspection, metadata copying, filtering, and artifact writing in the patched native model-package ABI. Rust owns the package manifest, provenance, topology inputs, and model-storage integration.

Package generation metadata

The package should expose the complete measured strategy menu and select mtp-suffix as its intended default once the benchmark gate passes:

{
  "generation": {
    "speculative_decoding": {
      "default": "mtp-suffix",
      "proposers": {
        "mtp": {
          "type": "native-mtp",
          "prediction_depth": 1,
          "layer_indices": [88]
        },
        "suffix": {
          "type": "ngram-suffix",
          "ngram_min": 5,
          "ngram_max": 32,
          "max_proposal_tokens": 48,
          "history_scope": "request"
        }
      },
      "strategies": {
        "mtp": {
          "type": "native-mtp",
          "proposer": "mtp"
        },
        "ngram-suffix": {
          "type": "ngram-suffix",
          "proposer": "suffix",
          "window_policy": {
            "default": "fixed",
            "initial_window": 16,
            "min_window": 1,
            "max_window": 32,
            "pipeline_depth": 1
          }
        },
        "mtp-suffix": {
          "type": "composite",
          "primary": "mtp",
          "extender": "suffix",
          "extension_policy": {
            "max_tokens": 48
          },
          "window_policy": {
            "default": "fixed",
            "initial_window": 16,
            "min_window": 1,
            "max_window": 32,
            "pipeline_depth": 1
          }
        }
      }
    }
  }
}

The numeric suffix and verification-window values above are starting candidates, not publication defaults. Tune at least:

• native MTP tokens: 1, 3, 5, 7;
• suffix ngram_min: 3, 5, 8;
• suffix ngram_max: 16, 32, 64;
• suffix proposal/extension budget: 8, 16, 32, 48; and
• verification window: 4, 8, 16, 32.

Keep pipeline_depth = 1 until this exact package and topology have evidence that a deeper pipeline improves throughput without destabilizing KV/session behavior.

If mtp-suffix does not beat both MTP-only and suffix-only on the coding/tool-loop gate, publish all strategies but select the winning measured strategy as the package default and record why.

Provenance

Record both upstream model components and the runtime proposer provenance:

• target repository and immutable revision;
• target GGUF shard names and checksums;
• MTPv2 repository and immutable revision;
• MTP SafeTensors checksum;
• llama.cpp converter revision;
• composition tool revision;
• suffix N-gram implementation revision, including Add Ngram Suffix Proposer #1037;
• tensor and generation-policy recipes; and
• final package artifact checksums.

The package manifest currently identifies one source model. Resolve this explicitly by either:

1. publishing a derived composite-GGUF source repository and referencing it; or
2. adding an optional, additive multi-source provenance structure to the package format.

Do not overload the existing single-source fields with ambiguous values.

Validation

Before publication:

• Composite GGUF opens successfully through ModelInfo.
• Target Q4 tensor contents/checksums match the original tensors.
• Target layers remain blk.0..87.
• MTP tensors appear only under blk.88.
• nextn_predict_layers = 1.
• Tokenizer identity matches the target.
• Package manifest reports layer_count = 89.
• Package manifest declares mtp, ngram-suffix, and mtp-suffix.
• The suffix proposer declares history_scope: request and passes package preflight.
• The composite declares native MTP as primary and suffix N-gram as extender.
• skippy-model-package preflight <package-dir> --stages 2 passes.
• Two-stage materialization places MTP tensors on the final stage.
• The package works without draft_model_path.
• Telemetry proves MTP prefix and suffix-tail attribution separately.
• Rejected suffix tails do not count as rejection of an accepted MTP prefix.
• Suffix history contains only prompt and target-committed tokens for the current request.
• Deterministic target output matches speculation-disabled target output.

Benchmark plan

Compare on the same package and topology:

• disabled;
• mtp;
• ngram-suffix;
• mtp-suffix; and
• optionally ngram-cache / mtp-cache as the bounded llama.cpp cache control.

Use coding-agent workloads covering:

• short coding completions;
• repository-context edits;
• file re-emission with localized changes;
• repeated structured configuration;
• tool-call loops and echoed tool output;
• repeated identifiers;
• long resident prefixes; and
• unique versus repeated prefixes.

Record:

• decode and end-to-end tokens/second;
• drafted, accepted, and rejected tokens by proposer source;
• MTP-prefix acceptance;
• suffix-tail acceptance conditional on an accepted MTP prefix;
• suffix hit/miss rate and lookup time;
• candidate length and accepted length distributions;
• MTP proposal and target verification time;
• TTFT and TPOT;
• peak memory by node;
• single-node and two-node split performance; and
• deterministic output parity.

Prefix-cache measurements remain separate: resident KV prefix reuse primarily improves repeated-prefix prefill/TTFT, while suffix N-gram is a request-local decode proposer and MTP predicts new decode tokens.

Initial performance hypothesis for single-stream coding on the Apple Silicon lab is approximately 1.5-2.3x target decode throughput for MTP alone, with mtp-suffix potentially improving repetitive coding/tool-loop workloads further. This is not an acceptance gate.

Follow-up quantization

Establish correctness with the BF16 MTP block first. Afterwards, produce package variants with only blk.88 quantized:

• Q8 MTP; and
• selective Q4 MTP.

Use skippy-quantize, not llama-quantize, for published Mesh artifacts. The existing target tensors must remain byte-identical. Report both TPS and MTP-prefix acceptance degradation for every MTP quantization; suffix-only behavior should remain invariant for the same committed history.

Relationship to #1416

Package construction and single-stream certification can happen before #1416.

After #1416 lands, add every package strategy to the scheduler's concurrency gate:

• multiple simultaneous MTP/suffix sessions;
• request-local suffix-history isolation;
• batched composite VerifyWindows;
• mixed prefill/decode scheduling;
• resident prefix-cache coexistence;
• KV-pressure and preemption behavior; and
• no regression at concurrency N=1.

The scheduler implementation remains outside this issue.

Non-goals

• Requantizing the complete 120B target.
• Replacing the existing non-MTP package.
• Requiring a local external sidecar path.
• Sharing N-gram history across requests or users.
• Treating suffix lookup as authoritative output.
• Implementing prefill/decode disaggregation.
• Implementing the Design: from-scratch iteration-level scheduler for concurrent staged serving #1416 scheduler.
• Claiming NVIDIA acceptance results as measured Mesh performance.

Deliverables

• Reproducible MTP-to-GGUF and composite-GGUF workflow.
• BF16-MTP composite GGUF or streaming package input.
• Self-contained MTPv2 layer package.
• Package-owned mtp, ngram-suffix, and mtp-suffix strategies.
• Multi-source/proposer provenance and checksums.
• Package preflight report.
• Single-node and two-node serving smoke tests.
• Deterministic target-output parity evidence.
• Coding/tool-loop benchmark across proposer and window settings.
• Lab memory measurements.
• Published Hugging Face package.
• Documentation and example configuration.
• Follow-up Design: from-scratch iteration-level scheduler for concurrent staged serving #1416 concurrency-gate task.

Acceptance criteria

• A new self-contained layer-package repository is published.
• The original Q4 target tensors remain byte-identical.
• The package builds without downloading or converting the complete BF16 target.
• Mesh discovers mtp, ngram-suffix, and mtp-suffix without an external model path.
• The composite uses native MTP as primary and Dan's request-local suffix proposer as extender.
• Two-stage preflight, materialization, and serving succeed.
• Deterministic output matches the speculation-disabled target.
• MTP prefix and suffix tail have separate acceptance/latency attribution.
• At least one package strategy improves coding decode TPS.
• The chosen default is the measured winner across the coding/tool-loop gate.
• Published artifacts include immutable provenance and checksums for both model sources and the generation-policy implementation.

References

• NVIDIA Nemotron 3 Super MTPv2
• NVIDIA Nemotron 3 Super BF16
• Unsloth Super GGUF
• Existing Mesh layer package
• Add Ngram Suffix Proposer #1037
• Design: from-scratch iteration-level scheduler for concurrent staged serving #1416