Skip to content

Commit 3f3a47e

Browse files
authored
Arm backend: Expand the Arm skills.md with advice for NGP performance (#24796)
Add a section in the SKILLS.md explaining the key principles how to optimise for performance for Arm Mali GPUs with Neural Engine. Signed-off-by: George Gekov <george.gekov@arm.com>
1 parent bdc7c35 commit 3f3a47e

1 file changed

Lines changed: 47 additions & 1 deletion

File tree

‎.claude/skills/arm/SKILL.md‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: arm
3-
description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, and node visitors. Not for development of other backends.
3+
description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, node visitors, performance optimizations. Not for development of other backends.
44
---
55

66
# Arm Backend Development
@@ -200,3 +200,49 @@ change. If execution requires approval or is blocked, report that rather than
200200
claiming it passed.
201201
For documentation-only changes, use focused documentation or skill validation
202202
and whitespace checks instead; explanations without file changes need no hook.
203+
204+
205+
# Optimising for Performance for the Arm backends.
206+
207+
## Arm VGF backend
208+
The Arm VGF backend targets Arm Mali GPU with Neural Engine technology(NX). The
209+
Arm Neural Engine technology provides silicon acceleration for the most intense
210+
mathematical computations in a neural network. Here are a few principles how to
211+
optimise for performance for Arm Mali GPUs with NX.
212+
213+
1. **Check that the model is quantized.**
214+
Inspect the TOSA graph for unintended float32 computation. For performance, prefer full
215+
static quantization of the model to 8x8 (8-bit activations and weights), or 16x8 (16-bit
216+
activations and 8-bit weights).
217+
Quantized inputs and outputs alone do not mean the internal model is
218+
quantized.
219+
220+
2. **Treat every NX/shader boundary as a memory cost.**
221+
When execution switches from NX to a shader and back, intermediate tensors
222+
must be materialized in shared memory. Budget for NX writing its output to external memory,
223+
the shader reading it and writing its result, and the following NX graph reading
224+
that result. These transfers will generate substantial DRAM traffic, especially
225+
if the tensor sizes are significant. Do enough useful work in the shader to
226+
justify the transfers and dispatches.
227+
228+
For example, if a bilinear RESIZE upsamples a tensor immediately before a
229+
shader, consider moving the RESIZE into that shader. The shader can often
230+
perform bilinear sampling efficiently. NX then writes the smaller tensor and
231+
the shader reads it, avoiding the expanded intermediate's write and read.
232+
Doubling both spatial dimensions makes that intermediate four times larger.
233+
Preserve the original resize coordinates and quantized rounding behavior.
234+
235+
3. **Avoid small NX graphs between shaders.**
236+
Model Converter can split NX work according to the source order around
237+
CUSTOM shader operations. A SLICE or a few elementwise operations between
238+
independent shaders can therefore create a separate DataGraph. Group
239+
independent shaders together and put their TOSA postprocessing afterwards
240+
when dependencies allow. Consecutive shaders do not need a DataGraph between
241+
them. Check the compiled VGF to confirm the small graphs disappeared.
242+
243+
4. **Fuse work to avoid intermediate tensors.**
244+
Consider combining related shaders and nearby operations such as
245+
PixelShuffle, resizing or blending. For example, a shader that warps two
246+
images can also blend them and write only the final image, avoiding two
247+
temporary images and a separate NX blend. Check the serializer's supported
248+
number of CUSTOM outputs before designing the fusion.

0 commit comments

Comments
 (0)