[Rules] Check 13 draft precision: process-wide ROCm opt-ins and whether measured real-workload AL can be evidence #3841

Ask

Check 13 (Draft runs as shipped) currently rules out two ROCm target-side optimizations for DeepSeek-V4.1-Flash because both knobs are process-wide in the pinned image and therefore also touch the DSpark draft. We would like to avoid carrying an unmerged upstream vLLM change purely as a gate-passing mechanism, and instead ask:

1. Will measured real-workload acceptance length — from the InferenceX GSM8K harness, with the real draft and real rejection sampling — be admitted as evidence under Check 13, given that CONTRIBUTING.md currently excludes only synthetic AgentX acceptance?
2. If not, can the rule state the acceptance criteria explicitly, so submitters know whether any measurement could ever satisfy it?

We are not asking for a blanket exemption, and we are not asking to ship a cheaper draft as a trick. We are asking whether the rule's own stated harm, when it is directly measured and found to be below run-to-run variation, can close the check.

What happened

• #3697 was rejected on Check 13: VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4 quantizes the DSpark draft's all-reduce activations to INT4.
• #3690 carries the same setting and is exposed to the same finding.

Why this is not draft-targeted precision hacking

Neither knob has draft/target granularity in the pinned image:

There is no supported way to apply either to the target only. So Check 13 does not merely constrain how we serve the draft — it removes a legitimate target-side gain entirely.

Data

All acceptance length (AL) figures below come from the eval server logs (eval_server_logs_*), i.e. the lm-eval GSM8K run with chat template, real draft and real rejection sampling. No synthetic acceptance is involved; we verified no synthetic-acceptance setting is present in any of these runs. Warmup and drain samples (AL > 4.05, where the batch is nearly empty and acceptance is trivially high) are excluded; every steady-state sample is listed.

Acceptance length, DeepSeek-V4.1-Flash + DSpark (k=5), MI355X, a8w4 target

Per-run detail:

Config attribution is taken from each run's own runtime Env: line, not from the recipe, so inherited settings are accounted for.

Two things we want to be precise about, rather than overstate:

• The AL delta is not zero. It is about −0.5%, in the direction the rule predicts. Our claim is only that it is smaller than baseline run-to-run variation (0.42× and 0.37× of the baseline sd, with baseline itself spanning 3.82–3.98 across 20 samples).
• Adding a4w4 on top of INT4 quick reduce does not compound the loss (3.891 → 3.893). If draft precision were the dominant term, stacking a second precision reduction on the draft's MoE should have shown up here.

GSM8K accuracy (5-shot, chat completions) — the baseline spread fully brackets the opt-in results:

We note that CONTRIBUTING.md is right that target-model evals do not measure this harm, and we are not resting the argument on GSM8K. We include it only to show there is no accuracy regression hiding behind the AL numbers.

On the rule's current wording

CONTRIBUTING.md says:

Matching an upstream recipe, passing evals, a claimed unchanged acceptance length (AL), or a newly measured AL curve does not exempt a submission that lowers draft precision. Synthetic AgentX acceptance is not evidence either way. Draft-precision changes shift the accuracy hit onto acceptance rate, which the target-model evals do not measure.

The stated harm is lost acceptance rate that evals do not measure. We agree with that mechanism. Our point is that acceptance rate is itself directly measurable, and the sentence excluding acceptance evidence names synthetic AgentX acceptance — which is indeed meaningless here. The numbers above are not synthetic: they are real-draft, real-rejection-sampling AL from the GSM8K eval harness, repeated across six evals and five sweeps.

As written, the clause covers both "a claimed unchanged AL" and "a newly measured AL curve", which makes the check unfalsifiable: there is no measurement a submitter could produce. If that is the intent, we would like it stated plainly, because it changes the rule from "do not degrade the draft" into "do not touch any global knob that reaches the draft", which is a much broader constraint and worth making explicit.

What we did upstream anyway

Because we accept the rule's intent, we implemented draft isolation in vLLM rather than arguing only on data:

vllm-project/vllm#59884 — "[Spec Decode][ROCm] Keep draft models at checkpoint precision under a4w4 and quantized quick reduce" (open, +178/−7)

It adds a draft_model_scope() applied once in initialize_model, covering construction, forward, and CUDA-graph capture on both model runners; FusedMoEConfig stops enabling a4w4 for draft MoE layers, and QuickAllReduce.should_quick_allreduce returns False inside a draft forward. Unit tests assert the gates (17 passed), and each gate's test fails when that gate is reverted.

We offer this as evidence of intent, and also as the reason for this issue. Using it to pass Check 13 means:

• it touches vllm/config, vllm/distributed, and the MoE config — core paths, for a benchmark gate;
• it is not merged, so it cannot be relied on; and
• even after it merges it must land in a pinned image before any submission can use it, since patching the pinned image is itself forbidden by the engine-patch rule.

So today the only compliant option is to drop a real target-side optimization, and the measured cost to the draft is below the noise floor of the metric the rule is protecting.

Suggested resolution

Any of these would unblock us; we have no preference beyond wanting the criteria written down:

1. Admit real-workload AL from the GSM8K harness as Check 13 evidence when the precision change comes from a knob with no target-only option upstream, with a stated threshold (e.g. AL within 1% and within baseline run-to-run variation, across ≥2 independent sweeps).
2. Keep Check 13 absolute, but say so explicitly — that no acceptance measurement can satisfy it — and confirm that the path forward is draft isolation landed in a pinned image.
3. Confirm that once vllm#59884 is merged and present in a pinned image, enabling these two knobs with draft isolation active is compliant, and state how reviewers should verify the scope is in effect.

Happy to run additional AL measurements — more sweeps, an AL-vs-concurrency curve, or an A/B on a single node with the two knobs as the only variable — if that would make the determination easier.

AI assistance was used to gather the run data and draft this issue. The data above was extracted from CI artifacts of the linked runs and is reproducible from them.