Skip to content

Commit 0c7f49d

Browse files
committed
Merge branch 'main' into jiacao/dsv4-mtp-dpa-ep8
Brings in fix(agentx): bump AIPerf to v1.0.1 (#2484) [65e0b1c]. # Conflicts: # perf-changelog.yaml
2 parents 8a428bf + d648c68 commit 0c7f49d

173 files changed

Lines changed: 12970 additions & 2410 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/codeowner-signoff-verify-prompt.md‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,20 @@ and read its `framework:`, `runner:`, and `image:` fields.
246246
the matching entry named.
247247
- N/A if the PR changes no master-config entries (state that in one line).
248248

249-
## Check 7 — No benchmark hacks that change the model architecture
249+
## Check 7 — No submissions for deprecated models or scenarios
250+
Read the current `MODELS.md` in the checked-out default branch. It is the source of
251+
truth for active and deprecated models, scenarios, and model-scenario combinations.
252+
For every benchmark configuration or recipe that the PR adds, changes, or re-enables,
253+
identify its model prefix and scenario, including fixed-sequence, agentic, single-node,
254+
and multi-node entries.
255+
- Use `date -u +%F` to establish the review date. Honor an effective date in
256+
`MODELS.md`, so a scheduled future deprecation is allowed until its stated date.
257+
- FAIL if the PR submits a model that is retired on the review date, a deprecated
258+
scenario, or a deprecated model-scenario combination. Name the model prefix,
259+
scenario, and the `MODELS.md` row or notice that prohibits it.
260+
- N/A if the PR adds, changes, or re-enables no benchmark configurations or recipes.
261+
262+
## Check 8 — No benchmark hacks that change the model architecture
250263
Verify from the PR diff (server args in `benchmarks/**` and master-config changes)
251264
that nothing alters the model architecture or reduces its FLOPs — e.g. `--hf-overrides`
252265
that skip the indexer every N layers on a model that doesn't natively support it,
@@ -261,7 +274,7 @@ production by accuracy-sensitive customers.
261274
- FAIL with the exact flag/value if architecture FLOPs are reduced without native
262275
model support. PASS in one line otherwise; N/A if the PR touches no server args.
263276

264-
## Check 8 — Speculative-decoding configs benchmark through chat templates
277+
## Check 9 — Speculative-decoding configs benchmark through chat templates
265278
If this PR adds or changes a speculative-decoding config (MTP / EAGLE / draft-model
266279
flags such as `--speculative-config`, `--speculative-algorithm`, `spec-decode`, config
267280
names ending in `-mtp`), verify the benchmark client exercises the model through its
@@ -271,7 +284,7 @@ completions) so the acceptance-length distribution matches real-world traffic.
271284
name the config and script line.
272285
- N/A if the PR has no speculative-decoding changes.
273286

274-
## Check 9 — No engine patches without a waiver
287+
## Check 10 — No engine patches without a waiver
275288
The pinned upstream image must run AS SHIPPED — the community must be able to
276289
reproduce the number from the released image. From the PR diff (scripts under
277290
`benchmarks/**`, master configs, workflow changes), scan for anything that modifies
@@ -300,7 +313,7 @@ Installing the benchmark harness and client-side deps (aiperf, eval tooling) is
300313
waiver not linked / waiver does not cover this patch).
301314
- N/A if the PR touches no benchmark scripts, images, or configs.
302315

303-
## Check 10 — Agentic spec-decode configs use the golden simulated acceptance length
316+
## Check 11 — Agentic spec-decode configs use the golden simulated acceptance length
304317
APPLICABILITY: this check covers AGENTIC-workload benchmark changes that enable
305318
speculative decoding. From the PR diff, identify configs that are BOTH:
306319
- agentic — scripts under `benchmarks/single_node/agentic/**`, multi-node recipes
@@ -342,12 +355,12 @@ Verify BOTH:
342355
additional detail section must state the source of the AL value (e.g. a pending
343356
golden-curve collection run); FAIL if it does not.
344357
- Also FAIL (as a benchmark hack) if simulated/synthetic-acceptance knobs appear on a
345-
NON-agentic spec-decode config, where Check 8's real-traffic AL standard applies —
358+
NON-agentic spec-decode config, where Check 9's real-traffic AL standard applies —
346359
unless the sign-off documents a sanctioned exception.
347360
- N/A if the PR has no agentic speculative-decoding changes (state that in one line).
348361

349362
## Verdict and output
350-
Decide PASS only if Checks 0-10 ALL pass (a check reported as `N/A` counts as a pass —
363+
Decide PASS only if Checks 0-11 ALL pass (a check reported as `N/A` counts as a pass —
351364
keep the `N/A — <reason>` row so the reviewer sees it was considered). Post EXACTLY ONE summary comment on
352365
PR #${PR_NUMBER} using `gh pr comment`. Start the comment with
353366
the hidden marker so reruns are identifiable:
@@ -373,8 +386,7 @@ single terse line either. Rules:
373386
restating the checklist, no hedging ("if X then maybe Y" — make the call). Link the
374387
run/recipe instead of describing it.
375388

376-
- If everything is to standard: post the verdict header + the eleven one-line rows
377-
(with the green run URL). Do NOT @-mention anyone on a pass.
389+
- If everything is to standard: post the verdict header + the twelve one-line rows
378390
- If anything is NOT to standard: the verdict header must be immediately followed by a
379391
line that @-mentions the sign-off author as `@${SIGNOFF_AUTHOR}`
380392
with the blocking summary. Then the per-check lines, each failing one led by its root

‎.github/workflows/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,27 @@ and eval jobs retain their canonical settings. Adding or removing the modifier
166166
restarts the active sweep. Fast-mode runs are not eligible for artifact reuse
167167
after merge.
168168

169+
## Trusted External-Fork Sweep Dispatch (PoC)
170+
171+
Public-fork `pull_request` workflows receive no repository secrets. For an
172+
external PR, the ordinary `run-sweep.yml` run therefore validates the
173+
changelog but does not fan out onto GPU runners. A maintainer with `write`,
174+
`maintain`, or `admin` permission can add any modifier labels first, then apply
175+
one primary sweep label to approve the PR's exact current head SHA.
176+
`trusted-external-sweep.yml` then dispatches `e2e-tests.yml` from `main`, pins
177+
both the approved head and GitHub's merge SHA, and runs the generated matrix
178+
with the trusted workflow's secrets.
179+
180+
The approval is revision-specific. A later push is not trusted automatically;
181+
remove and re-add the primary sweep label to approve the new SHA. The trusted
182+
dispatcher never checks out or executes PR code itself.
183+
184+
This proof of concept produces benchmark and evaluation artifacts through the
185+
End-to-End Tests workflow. Those runs are not yet eligible for
186+
`/reuse-sweep-run`, which currently accepts only `run-sweep.yml` runs. The PoC
187+
also fans out the selected matrix immediately; it does not reproduce
188+
`run-sweep.yml`'s canary-first sequencing.
189+
169190
## Reusing an Approved PR Full Sweep
170191

171192
`[skip-sweep]` skips PR benchmark setup only; changelog and reuse checks still

‎.github/workflows/benchmark-multinode-tmpl.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,16 @@ on:
138138
type: string
139139
required: false
140140
default: ""
141+
require-power:
142+
description: "Fail fixed-sequence result processing when GPU power is invalid"
143+
type: boolean
144+
required: false
145+
default: false
146+
power-producer-sha:
147+
description: "Expected srt-slurm producer git commit for power validation"
148+
type: string
149+
required: false
150+
default: ''
141151
# Agentic-coding inputs. Fixed-seq-len jobs leave these empty.
142152
scenario-type:
143153
description: "Scenario type (fixed-seq-len or agentic-coding)"
@@ -213,6 +223,8 @@ env:
213223
RUN_EVAL: ${{ inputs.run-eval }}
214224
EVAL_ONLY: ${{ inputs.eval-only }}
215225
EVAL_CONC: ${{ inputs.eval-conc }}
226+
REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
227+
POWER_PRODUCER_SHA: ${{ inputs.power-producer-sha }}
216228
# Agentic-coding env. Fixed-seq-len jobs leave these empty.
217229
SCENARIO_TYPE: ${{ inputs.scenario-type }}
218230
SCENARIO_SUBDIR: ${{ inputs.scenario-type == 'agentic-coding' && 'agentic/' || 'fixed_seq_len/' }}
@@ -358,6 +370,12 @@ jobs:
358370
env:
359371
RUNNER_TYPE: ${{ inputs.runner }}
360372
run: |
373+
# Launcher stamp supplies the producer SHA unless the
374+
# power-producer-sha input is set as a manual override
375+
if [ -z "$POWER_PRODUCER_SHA" ] && [ -f power-producer-sha.txt ]; then
376+
export POWER_PRODUCER_SHA="$(cat power-producer-sha.txt)"
377+
echo "POWER_PRODUCER_SHA derived from launcher stamp: $POWER_PRODUCER_SHA"
378+
fi
361379
# Process each result file
362380
for result_file in ${RESULT_FILENAME}_*.json; do
363381
if [ -f "$result_file" ]; then
@@ -386,6 +404,19 @@ jobs:
386404
name: bmk_${{ env.RESULT_FILENAME }}
387405
path: agg_${{ env.RESULT_FILENAME }}_*.json
388406

407+
- name: Upload power audit bundle
408+
if: ${{ always() && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' }}
409+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
410+
with:
411+
name: power_audit_${{ env.RESULT_FILENAME }}
412+
path: |
413+
${{ env.RESULT_FILENAME }}_*.json
414+
agg_${{ env.RESULT_FILENAME }}_*.json
415+
power_validation_${{ env.RESULT_FILENAME }}_*.json
416+
LOGS/power/**
417+
LOGS/*/results_*.json
418+
if-no-files-found: ignore
419+
389420
- name: Upload server logs
390421
if: always()
391422
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

‎.github/workflows/benchmark-tmpl.yml‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,11 @@ on:
129129
required: false
130130
type: string
131131
default: '3600'
132+
require-power:
133+
description: "Fail fixed-sequence result processing when GPU power is invalid"
134+
required: false
135+
type: boolean
136+
default: false
132137
agentx-fast:
133138
description: "Use one warmup request per AgentX lane and a 20-minute profile"
134139
required: false
@@ -179,6 +184,7 @@ env:
179184
KV_P2P_TRANSFER: ${{ inputs.kv-p2p-transfer }}
180185
TOTAL_CPU_DRAM_GB: ${{ inputs.total-cpu-dram-gb }}
181186
DURATION: ${{ inputs.duration }}
187+
REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
182188
AIPERF_EXPERIMENTAL_FAST: ${{ inputs.agentx-fast && '1' || '0' }}
183189
EVAL_LIMIT: ${{ inputs.eval-limit }}
184190
SWEBENCH_GEN_MODE: ${{ inputs.swebench-gen-mode }}
@@ -356,7 +362,26 @@ jobs:
356362
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
357363
with:
358364
name: ${{ inputs.eval-only && 'eval_gpu_metrics_' || 'gpu_metrics_' }}${{ env.RESULT_FILENAME }}
359-
path: gpu_metrics.csv
365+
path: |
366+
gpu_metrics.csv
367+
gpu_metrics_energy_start.csv
368+
gpu_metrics_energy_end.csv
369+
gpu_metrics_identity.json
370+
if-no-files-found: ignore
371+
372+
- name: Upload power audit bundle
373+
if: ${{ always() && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' }}
374+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
375+
with:
376+
name: power_audit_${{ env.RESULT_FILENAME }}
377+
path: |
378+
${{ env.RESULT_FILENAME }}.json
379+
agg_${{ env.RESULT_FILENAME }}.json
380+
gpu_metrics.csv
381+
gpu_metrics_energy_start.csv
382+
gpu_metrics_energy_end.csv
383+
gpu_metrics_identity.json
384+
power_validation_${{ env.RESULT_FILENAME }}.json
360385
if-no-files-found: ignore
361386

362387
- name: Upload eval results (if any)

‎.github/workflows/codeowner-signoff-verify.yml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,14 @@ name: CODEOWNER Sign-off Verify
2525
# 6. vLLM/SGLang configs run upstream images (hub.docker.com/u/vllm,
2626
# hub.docker.com/u/lmsysorg) on established hardware, and vLLM/SGLang
2727
# submissions land before additional frameworks (TRT-LLM, ATOM, ...).
28-
# 7. No benchmark hacks that change the model architecture / cut FLOPs.
29-
# 8. Speculative-decoding configs benchmark through chat templates.
30-
# 9. No patches to the inference engine / serving stack (the pinned image
28+
# 7. No submissions for models, scenarios, or model-scenario combinations
29+
# deprecated in MODELS.md.
30+
# 8. No benchmark hacks that change the model architecture / cut FLOPs.
31+
# 9. Speculative-decoding configs benchmark through chat templates.
32+
# 10. No patches to the inference engine / serving stack (the pinned image
3133
# runs as shipped) — the only exception is a patch covered by a
3234
# filled-out docs/waiver/<PR_NUMBER>.md waiver.
33-
# 10. Agentic spec-decode configs simulate acceptance at the committed golden
35+
# 11. Agentic spec-decode configs simulate acceptance at the committed golden
3436
# AL from golden_al_distribution/ (SGLang SGLANG_SIMULATE_ACC_* env vars,
3537
# vLLM synthetic rejection sampling in --speculative-config).
3638
# If any of those are not to standard, Claude posts a single comment that

‎.github/workflows/collectivex-sweep.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ on:
1010
description: "EP library to sweep — 'all' runs every EP backend in one matrix"
1111
type: choice
1212
default: all
13-
options: [all, deepep-v2, mori, uccl-ep, nccl-ep]
13+
options: [all, deepep-v2, mori, uccl-ep, nccl-ep, flashinfer-ep]
1414
only_sku:
1515
description: Restrict to one GHA runner pool; blank = all
1616
type: string
@@ -224,6 +224,11 @@ jobs:
224224
|| { echo "CollectiveX allocation cleanup failed; results may still be changing" >&2; exit 1; }
225225
python3 experimental/CollectiveX/summarize.py \
226226
--results-dir experimental/CollectiveX/results >> "$GITHUB_STEP_SUMMARY" || true
227+
# Second view over the same documents: latency answers how long a point took,
228+
# this answers how fast it moved its payload. Both are renderers over the staged
229+
# JSON and neither gates the leg, so both are `|| true`.
230+
python3 experimental/CollectiveX/bandwidth.py \
231+
--results-dir experimental/CollectiveX/results >> "$GITHUB_STEP_SUMMARY" || true
227232
shopt -s nullglob
228233
results=(experimental/CollectiveX/results/*.json)
229234
if [ "${#results[@]}" -eq 0 ]; then

0 commit comments

Comments
 (0)