@@ -246,7 +246,20 @@ and read its `framework:`, `runner:`, and `image:` fields.
246246 the matching entry named.
247247- N/A if the PR changes no master-config entries (state that in one line).
248248
249- ## Check 7 — No benchmark hacks that change the model architecture
249+ ## Check 7 — No submissions for deprecated models or scenarios
250+ Read the current ` MODELS.md ` in the checked-out default branch. It is the source of
251+ truth for active and deprecated models, scenarios, and model-scenario combinations.
252+ For every benchmark configuration or recipe that the PR adds, changes, or re-enables,
253+ identify its model prefix and scenario, including fixed-sequence, agentic, single-node,
254+ and multi-node entries.
255+ - Use ` date -u +%F ` to establish the review date. Honor an effective date in
256+ ` MODELS.md ` , so a scheduled future deprecation is allowed until its stated date.
257+ - FAIL if the PR submits a model that is retired on the review date, a deprecated
258+ scenario, or a deprecated model-scenario combination. Name the model prefix,
259+ scenario, and the ` MODELS.md ` row or notice that prohibits it.
260+ - N/A if the PR adds, changes, or re-enables no benchmark configurations or recipes.
261+
262+ ## Check 8 — No benchmark hacks that change the model architecture
250263Verify from the PR diff (server args in ` benchmarks/** ` and master-config changes)
251264that nothing alters the model architecture or reduces its FLOPs — e.g. ` --hf-overrides `
252265that skip the indexer every N layers on a model that doesn't natively support it,
@@ -261,7 +274,7 @@ production by accuracy-sensitive customers.
261274- FAIL with the exact flag/value if architecture FLOPs are reduced without native
262275 model support. PASS in one line otherwise; N/A if the PR touches no server args.
263276
264- ## Check 8 — Speculative-decoding configs benchmark through chat templates
277+ ## Check 9 — Speculative-decoding configs benchmark through chat templates
265278If this PR adds or changes a speculative-decoding config (MTP / EAGLE / draft-model
266279flags such as ` --speculative-config ` , ` --speculative-algorithm ` , ` spec-decode ` , config
267280names ending in ` -mtp ` ), verify the benchmark client exercises the model through its
@@ -271,7 +284,7 @@ completions) so the acceptance-length distribution matches real-world traffic.
271284 name the config and script line.
272285- N/A if the PR has no speculative-decoding changes.
273286
274- ## Check 9 — No engine patches without a waiver
287+ ## Check 10 — No engine patches without a waiver
275288The pinned upstream image must run AS SHIPPED — the community must be able to
276289reproduce the number from the released image. From the PR diff (scripts under
277290` benchmarks/** ` , master configs, workflow changes), scan for anything that modifies
@@ -300,7 +313,7 @@ Installing the benchmark harness and client-side deps (aiperf, eval tooling) is
300313 waiver not linked / waiver does not cover this patch).
301314- N/A if the PR touches no benchmark scripts, images, or configs.
302315
303- ## Check 10 — Agentic spec-decode configs use the golden simulated acceptance length
316+ ## Check 11 — Agentic spec-decode configs use the golden simulated acceptance length
304317APPLICABILITY: this check covers AGENTIC-workload benchmark changes that enable
305318speculative decoding. From the PR diff, identify configs that are BOTH:
306319- agentic — scripts under ` benchmarks/single_node/agentic/** ` , multi-node recipes
@@ -342,12 +355,12 @@ Verify BOTH:
342355 additional detail section must state the source of the AL value (e.g. a pending
343356 golden-curve collection run); FAIL if it does not.
344357- Also FAIL (as a benchmark hack) if simulated/synthetic-acceptance knobs appear on a
345- NON-agentic spec-decode config, where Check 8 's real-traffic AL standard applies —
358+ NON-agentic spec-decode config, where Check 9 's real-traffic AL standard applies —
346359 unless the sign-off documents a sanctioned exception.
347360- N/A if the PR has no agentic speculative-decoding changes (state that in one line).
348361
349362## Verdict and output
350- Decide PASS only if Checks 0-10 ALL pass (a check reported as ` N/A ` counts as a pass —
363+ Decide PASS only if Checks 0-11 ALL pass (a check reported as ` N/A ` counts as a pass —
351364keep the ` N/A — <reason> ` row so the reviewer sees it was considered). Post EXACTLY ONE summary comment on
352365PR #${PR_NUMBER} using ` gh pr comment ` . Start the comment with
353366the hidden marker so reruns are identifiable:
@@ -373,8 +386,7 @@ single terse line either. Rules:
373386 restating the checklist, no hedging ("if X then maybe Y" — make the call). Link the
374387 run/recipe instead of describing it.
375388
376- - If everything is to standard: post the verdict header + the eleven one-line rows
377- (with the green run URL). Do NOT @-mention anyone on a pass.
389+ - If everything is to standard: post the verdict header + the twelve one-line rows
378390- If anything is NOT to standard: the verdict header must be immediately followed by a
379391 line that @-mentions the sign-off author as ` @${SIGNOFF_AUTHOR} `
380392 with the blocking summary. Then the per-check lines, each failing one led by its root
0 commit comments