Skip to content

Commit 63a3e6d

Browse files
committed
Update (base update)
[ghstack-poisoned]
2 parents e6ff5fe + 4eb094b commit 63a3e6d

423 files changed

Lines changed: 26029 additions & 78759 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ci/docker/common/install_cuda_windows_cross_compile.sh‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,6 @@ INSTALL_DIR="${WINDOWS_CUDA_INSTALL_DIR:-/opt/cuda-windows}"
1414
# Mapping of CUDA versions to their corresponding driver versions for Windows installers
1515
# Source: https://developer.nvidia.com/cuda-toolkit-archive
1616
declare -A CUDA_DRIVER_MAP=(
17-
["12.6"]="12.6.3:561.17"
1817
["12.8"]="12.8.1:572.61"
1918
["12.9"]="12.9.1:576.57"
2019
["13.0"]="13.0.2:"

‎.ci/scripts/cuda_benchmark.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -588,7 +588,7 @@ def to_v3_format(
588588
workflow_run_id: GitHub workflow run ID
589589
workflow_run_url: GitHub workflow run URL
590590
gpu_name: GPU device name (e.g., "Tesla V100", "A100")
591-
cuda_driver_version: CUDA driver version (e.g., "12.6", "535.104.05")
591+
cuda_driver_version: CUDA driver version (e.g., "13.0", "535.104.05")
592592
593593
Returns:
594594
List of benchmark records in v3 format
@@ -873,7 +873,7 @@ def main():
873873
"--cuda_driver_version",
874874
type=str,
875875
default=None,
876-
help="CUDA driver version (e.g., '12.6', '535.104.05') - optional for v3 format",
876+
help="CUDA driver version (e.g., '13.0', '535.104.05') - optional for v3 format",
877877
)
878878
parser.add_argument("--verbose", action="store_true", help="Print verbose output")
879879

‎.ci/scripts/test_espressif_idf_component.sh‎

Lines changed: 312 additions & 36 deletions
Large diffs are not rendered by default.

‎.ci/scripts/test_lora.sh‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,18 @@ import re
4646
import sys
4747
4848
text = pathlib.Path(sys.argv[1]).read_text()
49+
# Under 8da4w, the base model's first token is a near-tie between ":\n" and ":"
50+
# that tips with small numeric changes, such as RMSNorm and QK-norm moving into
51+
# XNNPACK. Accept both observed continuations.
4952
pattern = re.compile(
50-
r"^<\|im_start\|>user Calculate 15% of 80\?<\|im_end\|><\|im_start\|>assistant:\n"
51-
r"(?:<think>\n)+"
53+
r"^<\|im_start\|>user Calculate 15% of 80\?<\|im_end\|><\|im_start\|>assistant:"
54+
r"(?:\n(?:<think>\n)+| me\n)"
5255
r"Okay, so I need to calculate 15% of 80\.",
5356
re.MULTILINE,
5457
)
55-
sys.exit(0 if pattern.match(text) else 1)
58+
if not pattern.match(text):
59+
print(f"Expected a match for: {pattern.pattern}")
60+
sys.exit(1)
5661
PY
5762
}
5863

@@ -174,9 +179,6 @@ else
174179
fi
175180

176181
### QUANTIZATION & PROGRAM DATA SEPARATION ###
177-
EXPECTED_QUANT_PREFIX="<|im_start|>user Calculate 15% of 80?<|im_end|><|im_start|>assistant:
178-
<think>
179-
Okay, so I need to calculate 15% of 80."
180182
EXPECTED_QUANT_LORA_PREFIX="
181183
<|im_start|>user Calculate 15% of 80?<|im_end|><|im_start|>assistant
182184
To calculate 15% of 80, we can multiply 80 by 15/100.
@@ -238,11 +240,9 @@ NOW=$(date +"%H:%M:%S")
238240
echo "Finished at ${NOW}"
239241
RESULT=$(cat result.txt)
240242
if matches_base_response_prefix result.txt; then
241-
echo "Expected result prefix: ${EXPECTED_QUANT_PREFIX}"
242243
echo "Actual result: ${RESULT}"
243244
echo "Test 3: Success"
244245
else
245-
echo "Expected result prefix: ${EXPECTED_QUANT_PREFIX}"
246246
echo "Actual result: ${RESULT}"
247247
echo "Test 3: Failure; results not the same"
248248
cleanup_files

‎.ci/scripts/test_lora_multimethod.sh‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,18 @@ import re
4646
import sys
4747
4848
text = pathlib.Path(sys.argv[1]).read_text()
49+
# Under 8da4w, the base model's first token is a near-tie between ":\n" and ":"
50+
# that tips with small numeric changes, such as RMSNorm and QK-norm moving into
51+
# XNNPACK. Accept both observed continuations.
4952
pattern = re.compile(
50-
r"^<\|im_start\|>user Calculate 15% of 80\?<\|im_end\|><\|im_start\|>assistant:\n"
51-
r"(?:<think>\n)+"
53+
r"^<\|im_start\|>user Calculate 15% of 80\?<\|im_end\|><\|im_start\|>assistant:"
54+
r"(?:\n(?:<think>\n)+| me\n)"
5255
r"Okay, so I need to calculate 15% of 80\.",
5356
re.MULTILINE,
5457
)
55-
sys.exit(0 if pattern.match(text) else 1)
58+
if not pattern.match(text):
59+
print(f"Expected a match for: {pattern.pattern}")
60+
sys.exit(1)
5661
PY
5762
}
5863

@@ -107,10 +112,6 @@ EXPECTED_LORA_PREFIX="
107112
<|im_start|>user Calculate 15% of 80?<|im_end|><|im_start|>assistant
108113
To calculate 15% of 80"
109114

110-
EXPECTED_BASE_PREFIX="<|im_start|>user Calculate 15% of 80?<|im_end|><|im_start|>assistant:
111-
<think>
112-
Okay, so I need to calculate 15% of 80."
113-
114115
### TEST 1: Run lora_forward method ###
115116
NOW=$(date +"%H:%M:%S")
116117
echo "Test 1: Multimethod lora_forward. Starting at ${NOW}"
@@ -151,7 +152,6 @@ if matches_base_response_prefix result_base.txt; then
151152
echo "Test 2 (base_forward): Success"
152153
else
153154
echo "Test 2 (base_forward): Failure"
154-
echo "Expected result prefix: ${EXPECTED_BASE_PREFIX}"
155155
echo "Actual result: ${RESULT}"
156156
cleanup_files
157157
exit 1

‎.ci/scripts/tests/test_cu134_dependencies.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ def test_all_install_steps_preserve_exact_cu134_selection(self):
103103
)
104104

105105
def test_other_cuda_trains_keep_existing_pins(self):
106-
for cuda in ((12, 6), (13, 0), (13, 2)):
106+
for cuda in ((13, 0), (13, 2)):
107107
for machine in ("x86_64", "aarch64"):
108108
with self.subTest(cuda=cuda, machine=machine):
109109
core, local, domains, examples = self.install_commands(
@@ -159,7 +159,7 @@ def test_cpu_release_dependencies_are_opt_in(self):
159159
)
160160

161161
def test_cpu_release_option_keeps_cuda_pins(self):
162-
for cuda in ((12, 6), (13, 0), (13, 2), (13, 4)):
162+
for cuda in ((13, 0), (13, 2), (13, 4)):
163163
for machine in ("x86_64", "aarch64"):
164164
with self.subTest(cuda=cuda, machine=machine):
165165
self.assertEqual(

‎.ci/scripts/tests/test_cuda_workflow.py‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,7 @@ def _model_quant(entry):
3434
class CudaWorkflowTest(unittest.TestCase):
3535
def test_build_matrix_preserves_existing_cuda_versions(self):
3636
job = WORKFLOW["jobs"]["test-cuda-builds"]
37-
self.assertEqual(
38-
job["strategy"]["matrix"]["cuda-version"], ["12.6", "13.0", "13.4"]
39-
)
37+
self.assertEqual(job["strategy"]["matrix"]["cuda-version"], ["13.0", "13.4"])
4038
self.assertEqual(job["with"]["gpu-arch-version"], "${{ matrix.cuda-version }}")
4139

4240
def test_cuda_builds_take_the_node_driver_on_an_unpinned_v3(self):
@@ -58,9 +56,7 @@ def test_cuda_runtime_update_precedes_build_and_propagates_failure(self):
5856
source() { printf 'BUILD %s\n' "$*"; printf 'LIBRARY PATH %s\n' "$LD_LIBRARY_PATH"; }
5957
"""
6058
for version, conda_status in (
61-
(version, status)
62-
for version in ("12.6", "13.0", "13.4")
63-
for status in (0, 1)
59+
(version, status) for version in ("13.0", "13.4") for status in (0, 1)
6460
):
6561
with self.subTest(version=version, conda_status=conda_status):
6662
result = subprocess.run(

‎.ci/scripts/wheel/cuda_arch_list.sh‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -41,29 +41,6 @@ _cuda_arch_aarch64_cu130="8.0 9.0 10.0 11.0 12.0"
4141
_cuda_arch_aarch64_cu132="${_cuda_arch_aarch64_cu130}"
4242
_cuda_arch_aarch64_cu134="${_cuda_arch_aarch64_cu130}"
4343

44-
# The older CUDA train.
45-
#
46-
# The two architectures do not carry identical lists, because each covers what the published PyTorch
47-
# build for that architecture covers, and those differ. Matching them to each other instead would mean
48-
# advertising a GPU on one architecture that PyTorch cannot serve there.
49-
#
50-
# The smaller embedded modules are deliberately absent, with one exception. An embedded-only
51-
# architecture in a generic wheel would advertise a device the row cannot otherwise serve, since
52-
# those devices also need the CUDA, TensorRT and PyTorch pinned by their own software release
53-
# rather than the ones a generic wheel resolves.
54-
#
55-
# 8.7 is that exception. This is the only row whose CUDA major matches what that module's software
56-
# release ships, and the wheel declares no PyTorch, so the user supplies the build that carries
57-
# their architecture. Omitting it does not protect them from a bad pairing, it only removes the
58-
# device code they need.
59-
#
60-
# The floor is 8.0 rather than the oldest architecture PyTorch still carries. One of these sources compiles
61-
# an integer matrix-multiply path only at 8.0 and newer, so an older architecture would get a delegate that
62-
# loads, runs most models, and fails on one that needs that operator. Claiming hardware the delegate only
63-
# partly serves is the same problem the embedded modules have, so the row leaves it out for the same reason.
64-
_cuda_arch_x86_64_cu126="8.0 8.6 8.9 9.0"
65-
_cuda_arch_aarch64_cu126="8.0 8.7 9.0"
66-
6744
# A CUDA train with no architecture list would leave the build detecting the builder's GPU, which is
6845
# the failure this file exists to prevent. Adding a train to the release matrix without adding its
6946
# architectures should fail loudly rather than silently produce a single-GPU wheel.
@@ -103,7 +80,6 @@ executorch_cuda_arch_list() {
10380
case "${machine}" in
10481
aarch64 | arm64)
10582
case "${train}" in
106-
126) printf '%s' "${_cuda_arch_aarch64_cu126}" ;;
10783
130) printf '%s' "${_cuda_arch_aarch64_cu130}" ;;
10884
132) printf '%s' "${_cuda_arch_aarch64_cu132}" ;;
10985
134) printf '%s' "${_cuda_arch_aarch64_cu134}" ;;
@@ -112,7 +88,6 @@ executorch_cuda_arch_list() {
11288
;;
11389
x86_64)
11490
case "${train}" in
115-
126) printf '%s' "${_cuda_arch_x86_64_cu126}" ;;
11691
130) printf '%s' "${_cuda_arch_x86_64_cu130}" ;;
11792
132) printf '%s' "${_cuda_arch_x86_64_cu132}" ;;
11893
134) printf '%s' "${_cuda_arch_x86_64_cu134}" ;;

‎.ci/scripts/wheel/test_shared_libraries.py‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@
107107
"aoti_torch_index_put_out",
108108
"aoti_torch_cuda_scatter_reduce_two_out",
109109
"aoti_torch_cuda_scatter_src_out",
110+
"aoti_torch_cuda_masked_select",
110111
)
111112

112113
_QUANTIZED_KERNEL_SYMBOLS = (
@@ -1217,7 +1218,7 @@ def _wheel_cuda_train() -> str:
12171218
"""The CUDA train the installed wheel was built for, or "" for a CPU wheel.
12181219
12191220
Read from the local version segment, which is the only place the wheel states what
1220-
it was built for. `1.5.0+cu126` gives "126".
1221+
it was built for. `1.5.0+cu132` gives "132".
12211222
"""
12221223
local = importlib.metadata.version("executorch").partition("+")[2]
12231224
return local[2:] if local.startswith("cu") else ""
@@ -1360,7 +1361,7 @@ def _resolve_required(required):
13601361
),
13611362
# The CUDA components. Required exactly when the wheel says it is a CUDA wheel,
13621363
# which is decided at check time rather than here: a fixed False meant a wheel
1363-
# tagged +cu126 carrying no CUDA library at all passed every check in this file,
1364+
# tagged +cu132 carrying no CUDA library at all passed every check in this file,
13641365
# while a fixed True would fail every CPU wheel. The marker is the string these
13651366
# rows are keyed on below.
13661367
(
@@ -1457,7 +1458,7 @@ def test_each_component_has_one_owner() -> None:
14571458
shipped = {path.name for path in _shipped_shared_objects(_installed_package_dir())}
14581459
for what, symbols, owner, required in _OWNED_COMPONENTS:
14591460
# Resolved here rather than in the table, because it depends on the installed
1460-
# wheel. A fixed False let a wheel tagged +cu126 ship with no CUDA library at
1461+
# wheel. A fixed False let a wheel tagged +cu132 ship with no CUDA library at
14611462
# all and still pass, which is the whole point of the conditional rows.
14621463
required = _resolve_required(required)
14631464
present = any(name.startswith(owner) for name in shipped)

‎.claude/skills/arm/SKILL.md‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: arm
3-
description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, and node visitors. Not for development of other backends.
3+
description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, node visitors, performance optimizations. Not for development of other backends.
44
---
55

66
# Arm Backend Development
@@ -200,3 +200,49 @@ change. If execution requires approval or is blocked, report that rather than
200200
claiming it passed.
201201
For documentation-only changes, use focused documentation or skill validation
202202
and whitespace checks instead; explanations without file changes need no hook.
203+
204+
205+
# Optimising for Performance for the Arm backends.
206+
207+
## Arm VGF backend
208+
The Arm VGF backend targets Arm Mali GPU with Neural Engine technology(NX). The
209+
Arm Neural Engine technology provides silicon acceleration for the most intense
210+
mathematical computations in a neural network. Here are a few principles how to
211+
optimise for performance for Arm Mali GPUs with NX.
212+
213+
1. **Check that the model is quantized.**
214+
Inspect the TOSA graph for unintended float32 computation. For performance, prefer full
215+
static quantization of the model to 8x8 (8-bit activations and weights), or 16x8 (16-bit
216+
activations and 8-bit weights).
217+
Quantized inputs and outputs alone do not mean the internal model is
218+
quantized.
219+
220+
2. **Treat every NX/shader boundary as a memory cost.**
221+
When execution switches from NX to a shader and back, intermediate tensors
222+
must be materialized in shared memory. Budget for NX writing its output to external memory,
223+
the shader reading it and writing its result, and the following NX graph reading
224+
that result. These transfers will generate substantial DRAM traffic, especially
225+
if the tensor sizes are significant. Do enough useful work in the shader to
226+
justify the transfers and dispatches.
227+
228+
For example, if a bilinear RESIZE upsamples a tensor immediately before a
229+
shader, consider moving the RESIZE into that shader. The shader can often
230+
perform bilinear sampling efficiently. NX then writes the smaller tensor and
231+
the shader reads it, avoiding the expanded intermediate's write and read.
232+
Doubling both spatial dimensions makes that intermediate four times larger.
233+
Preserve the original resize coordinates and quantized rounding behavior.
234+
235+
3. **Avoid small NX graphs between shaders.**
236+
Model Converter can split NX work according to the source order around
237+
CUSTOM shader operations. A SLICE or a few elementwise operations between
238+
independent shaders can therefore create a separate DataGraph. Group
239+
independent shaders together and put their TOSA postprocessing afterwards
240+
when dependencies allow. Consecutive shaders do not need a DataGraph between
241+
them. Check the compiled VGF to confirm the small graphs disappeared.
242+
243+
4. **Fuse work to avoid intermediate tensors.**
244+
Consider combining related shaders and nearby operations such as
245+
PixelShuffle, resizing or blending. For example, a shader that warps two
246+
images can also blend them and write only the final image, avoiding two
247+
temporary images and a separate NX blend. Check the serializer's supported
248+
number of CUSTOM outputs before designing the fusion.

0 commit comments

Comments
 (0)