|
1 | 1 | --- |
2 | 2 | name: arm |
3 | | -description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, and node visitors. Not for development of other backends. |
| 3 | +description: Develop, debug, or explain the Arm backend in backends/arm/ and its community-facing examples in examples/arm/. Covers partitioning, TOSA lowering, VGF, Ethos-U, quantization, graph passes, memory layout, node visitors, performance optimizations. Not for development of other backends. |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | # Arm Backend Development |
@@ -200,3 +200,49 @@ change. If execution requires approval or is blocked, report that rather than |
200 | 200 | claiming it passed. |
201 | 201 | For documentation-only changes, use focused documentation or skill validation |
202 | 202 | and whitespace checks instead; explanations without file changes need no hook. |
| 203 | + |
| 204 | + |
| 205 | +# Optimising for Performance for the Arm backends. |
| 206 | + |
| 207 | +## Arm VGF backend |
| 208 | +The Arm VGF backend targets Arm Mali GPU with Neural Engine technology(NX). The |
| 209 | +Arm Neural Engine technology provides silicon acceleration for the most intense |
| 210 | +mathematical computations in a neural network. Here are a few principles how to |
| 211 | +optimise for performance for Arm Mali GPUs with NX. |
| 212 | + |
| 213 | +1. **Check that the model is quantized.** |
| 214 | + Inspect the TOSA graph for unintended float32 computation. For performance, prefer full |
| 215 | + static quantization of the model to 8x8 (8-bit activations and weights), or 16x8 (16-bit |
| 216 | + activations and 8-bit weights). |
| 217 | + Quantized inputs and outputs alone do not mean the internal model is |
| 218 | + quantized. |
| 219 | + |
| 220 | +2. **Treat every NX/shader boundary as a memory cost.** |
| 221 | + When execution switches from NX to a shader and back, intermediate tensors |
| 222 | + must be materialized in shared memory. Budget for NX writing its output to external memory, |
| 223 | + the shader reading it and writing its result, and the following NX graph reading |
| 224 | + that result. These transfers will generate substantial DRAM traffic, especially |
| 225 | + if the tensor sizes are significant. Do enough useful work in the shader to |
| 226 | + justify the transfers and dispatches. |
| 227 | + |
| 228 | + For example, if a bilinear RESIZE upsamples a tensor immediately before a |
| 229 | + shader, consider moving the RESIZE into that shader. The shader can often |
| 230 | + perform bilinear sampling efficiently. NX then writes the smaller tensor and |
| 231 | + the shader reads it, avoiding the expanded intermediate's write and read. |
| 232 | + Doubling both spatial dimensions makes that intermediate four times larger. |
| 233 | + Preserve the original resize coordinates and quantized rounding behavior. |
| 234 | + |
| 235 | +3. **Avoid small NX graphs between shaders.** |
| 236 | + Model Converter can split NX work according to the source order around |
| 237 | + CUSTOM shader operations. A SLICE or a few elementwise operations between |
| 238 | + independent shaders can therefore create a separate DataGraph. Group |
| 239 | + independent shaders together and put their TOSA postprocessing afterwards |
| 240 | + when dependencies allow. Consecutive shaders do not need a DataGraph between |
| 241 | + them. Check the compiled VGF to confirm the small graphs disappeared. |
| 242 | + |
| 243 | +4. **Fuse work to avoid intermediate tensors.** |
| 244 | + Consider combining related shaders and nearby operations such as |
| 245 | + PixelShuffle, resizing or blending. For example, a shader that warps two |
| 246 | + images can also blend them and write only the final image, avoiding two |
| 247 | + temporary images and a separate NX blend. Check the serializer's supported |
| 248 | + number of CUSTOM outputs before designing the fusion. |
0 commit comments