Skip to content

Commit fcacda3

Browse files
authored
Fix the Windows Voxtral Realtime CI test (pytorch#24270)
The Windows Voxtral Realtime test could never pass, for two reasons. Both came in with pytorch#17993, which added `-Mode` to `test_model_e2e_windows.ps1` with streaming as the default and switched the expected word to "Quilter", without updating the workflow that calls the script. - The workflow exports the model in offline mode but called the test without `-Mode`, so the script ran the runner with `--streaming`. The runner refuses `--streaming` for a model exported without it and aborts. Release builds compile out the log, so the job only showed exit code 0xC0000409. The mode now lives in one place: a `mode` key on the Voxtral Realtime row of both matrices, passed by the export and the test step alike. Other rows have no key and pass an empty mode, which both scripts treat as before. - The test looked for "Quilter" in a transcript of a poem recording that does not contain that word. "Quilter" comes from the LibriSpeech clip the Linux test uses. The test now uses the LibriSpeech clip the Windows Parakeet test already downloads (saved as `test_audio.wav` instead of `poem.wav`) and looks for "Phoebe", as the MLX Voxtral Realtime test does with the same clip. A second commit adds `export_model_artifact.sh` and `test_model_e2e_windows.ps1` to the workflow's paths filter and to the export and test job conditions, as cuda.yml does for its own scripts. Before, a change that only touched those scripts ran no Windows model job. Testing: on Linux CUDA, a Release runner with logging off reproduces the Windows abort with the old flags, and with the new ones transcribes the clip and prints "Phoebe". This branch lives in the main repository so the Windows CUDA jobs run on this PR; on the previous head all six Windows model tests passed, including Voxtral Realtime with "Phoebe".
1 parent ffbc853 commit fcacda3

2 files changed

Lines changed: 13 additions & 9 deletions

File tree

‎.ci/scripts/test_model_e2e_windows.ps1‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,12 +89,12 @@ switch ($HfModel) {
8989
$runnerTarget = "voxtral_realtime_runner"
9090
$runnerPath = "voxtral_realtime"
9191
$runnerPreset = "voxtral-realtime-cuda"
92-
$expectedOutput = "Quilter"
92+
$expectedOutput = "Phoebe"
9393
$preprocessor = "preprocessor.pte"
9494
$tokenizerUrl = ""
9595
$tokenizerFile = "tekken.json"
96-
$audioUrl = "https://github.com/voxserv/audio_quality_testing_samples/raw/refs/heads/master/testaudio/16000/test01_20s.wav"
97-
$audioFile = "poem.wav"
96+
$audioUrl = "https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav"
97+
$audioFile = "test_audio.wav"
9898
}
9999
"facebook/dinov2-small-imagenet1k-1-layer" {
100100
$runnerTarget = "dinov2_runner"

‎.github/workflows/cuda-windows.yml‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ on:
1717
- backends/cuda/**
1818
- backends/aoti/**
1919
- extension/cuda/**
20+
- .ci/scripts/export_model_artifact.sh
21+
- .ci/scripts/test_model_e2e_windows.ps1
2022
# Only unittest-cuda-runtime-windows runs for these alone; the other jobs
2123
# filter on their own paths and stay skipped.
2224
- .ci/scripts/test_cuda_runtime_windows.ps1
@@ -61,6 +63,8 @@ jobs:
6163
contains(needs.changed-files.outputs.changed-files, 'backends/aoti') ||
6264
contains(needs.changed-files.outputs.changed-files, 'extension/cuda') ||
6365
contains(needs.changed-files.outputs.changed-files, '.github/workflows/cuda-windows.yml') ||
66+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/export_model_artifact.sh') ||
67+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test_model_e2e_windows.ps1') ||
6468
needs.run-decision.outputs.is-full-run == 'true'
6569
)
6670
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
@@ -92,6 +96,7 @@ jobs:
9296
- model_repo: "mistralai"
9397
model_name: "Voxtral-Mini-4B-Realtime-2602"
9498
quant: "quantized-int4-tile-packed"
99+
mode: "vr-offline"
95100
- model_repo: "facebook"
96101
model_name: "dinov2-small-imagenet1k-1-layer"
97102
quant: "non-quantized"
@@ -147,11 +152,7 @@ jobs:
147152
echo "::endgroup::"
148153
fi
149154
150-
VR_MODE=""
151-
if [ "${{ matrix.model_name }}" = "Voxtral-Mini-4B-Realtime-2602" ]; then
152-
VR_MODE="vr-offline"
153-
fi
154-
source .ci/scripts/export_model_artifact.sh cuda-windows "${{ matrix.model_repo }}/${{ matrix.model_name }}" "${{ matrix.quant }}" "${RUNNER_ARTIFACT_DIR}" "${VR_MODE}"
155+
source .ci/scripts/export_model_artifact.sh cuda-windows "${{ matrix.model_repo }}/${{ matrix.model_name }}" "${{ matrix.quant }}" "${RUNNER_ARTIFACT_DIR}" "${{ matrix.mode }}"
155156
156157
test-model-cuda-windows-e2e:
157158
name: test-model-cuda-windows-e2e
@@ -168,6 +169,8 @@ jobs:
168169
contains(needs.changed-files.outputs.changed-files, 'backends/aoti') ||
169170
contains(needs.changed-files.outputs.changed-files, 'extension/cuda') ||
170171
contains(needs.changed-files.outputs.changed-files, '.github/workflows/cuda-windows.yml') ||
172+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/export_model_artifact.sh') ||
173+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test_model_e2e_windows.ps1') ||
171174
needs.run-decision.outputs.is-full-run == 'true'
172175
)
173176
uses: pytorch/test-infra/.github/workflows/windows_job.yml@main
@@ -195,6 +198,7 @@ jobs:
195198
- model_repo: "mistralai"
196199
model_name: "Voxtral-Mini-4B-Realtime-2602"
197200
quant: "quantized-int4-tile-packed"
201+
mode: "vr-offline"
198202
- model_repo: "facebook"
199203
model_name: "dinov2-small-imagenet1k-1-layer"
200204
quant: "non-quantized"
@@ -228,7 +232,7 @@ jobs:
228232
throw 'RUNNER_ARTIFACT_DIR is empty. Ensure download-artifact is configured for windows_job.yml.'
229233
}
230234
231-
.ci/scripts/test_model_e2e_windows.ps1 -Device cuda-windows -HfModel '${{ matrix.model_repo }}/${{ matrix.model_name }}' -QuantName '${{ matrix.quant }}' -ModelDir \$artifactDir -ExpectedCudaVersion '13.2'
235+
.ci/scripts/test_model_e2e_windows.ps1 -Device cuda-windows -HfModel '${{ matrix.model_repo }}/${{ matrix.model_name }}' -QuantName '${{ matrix.quant }}' -ModelDir \$artifactDir -ExpectedCudaVersion '13.2' -Mode '${{ matrix.mode }}'
232236
}"
233237
234238
# The counterpart of cuda.yml's unittest-cuda-runtime on Windows. The Windows GPU runners

0 commit comments

Comments
 (0)