Per-channel int8 quantized linear (#24397) by pssrawat · Pull Request #24397 · pytorch/executorch

pssrawat · 2026-11-01T04:00:53Z

Summary:

cortex_m::quantized_linear supports only per-tensor weights: it calls arm_fully_connected_s8 with a single requantization multiplier and shift. This adds per-channel linear weights.

With CortexMQuantizer.set_node_target (previous diff), set_node_target(torch.ops.aten.linear.default, INT8_PER_CHANNEL_CONFIG) gives per-channel int8 linear weights; without it, linear layers keep per_tensor_config, so existing exports are unchanged. CortexMLinearCheck rejects per-channel weights that are not quantized per output feature (axis 0, or unset as the convolution check allows), and the lowering checks that it gets one weight scale per output feature, since the kernel indexes its multipliers by output feature.

Per-channel linear weights lower to a new cortex_m::quantized_linear_per_channel. It matches quantized_linear but takes int32 multiplier and shift tensors, as the convolution ops do, and calls arm_fully_connected_per_channel_s8. CMSIS-NN takes int32 arrays, so tensors avoid any runtime conversion. int[] lists would also cost about 20 bytes per output feature in the program, where a tensor costs 4. The two C++ entry points share one implementation that takes the CMSIS-NN call; quantized_linear is unchanged. The Python reference requantizes each output feature with its own parameters.

This diff was authored with Claude Code.

Differential Revision: D126929702

pytorch-bot · 2026-11-01T04:00:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24397

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit bd2a5a5 with merge base dd38e81 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-11-01T04:01:32Z

@pssrawat has exported this pull request. If you are a Meta employee, you can view the originating Diff in D126929702.

Summary: Pull Request resolved: pytorch#24397 `cortex_m::quantized_linear` supports only per-tensor weights: it calls `arm_fully_connected_s8` with a single requantization multiplier and shift. This adds per-channel linear weights. With `CortexMQuantizer.set_node_target` (previous diff), `set_node_target(torch.ops.aten.linear.default, INT8_PER_CHANNEL_CONFIG)` gives per-channel int8 linear weights; without it, linear layers keep `per_tensor_config`, so existing exports are unchanged. `CortexMLinearCheck` rejects per-channel weights that are not quantized per output feature (axis 0, or unset as the convolution check allows), and the lowering checks that it gets one weight scale per output feature, since the kernel indexes its multipliers by output feature. Per-channel linear weights lower to a new `cortex_m::quantized_linear_per_channel`. It matches `quantized_linear` but takes int32 multiplier and shift tensors, as the convolution ops do, and calls `arm_fully_connected_per_channel_s8`. CMSIS-NN takes int32 arrays, so tensors avoid any runtime conversion. `int[]` lists would also cost about 20 bytes per output feature in the program, where a tensor costs 4. The two C++ entry points share one implementation that takes the CMSIS-NN call; `quantized_linear` is unchanged. The Python reference requantizes each output feature with its own parameters. This diff was authored with Claude Code. Differential Revision: D126929702

Summary: `cortex_m::quantized_linear` supports only per-tensor weights: it calls `arm_fully_connected_s8` with a single requantization multiplier and shift. This adds per-channel linear weights. With `CortexMQuantizer.set_node_target` (previous diff), `set_node_target(torch.ops.aten.linear.default, INT8_PER_CHANNEL_CONFIG)` gives per-channel int8 linear weights; without it, linear layers keep `per_tensor_config`, so existing exports are unchanged. `CortexMLinearCheck` rejects per-channel weights that are not quantized per output feature (axis 0, or unset as the convolution check allows), and the lowering checks that it gets one weight scale per output feature, since the kernel indexes its multipliers by output feature. Per-channel linear weights lower to a new `cortex_m::quantized_linear_per_channel`. It matches `quantized_linear` but takes int32 multiplier and shift tensors, as the convolution ops do, and calls `arm_fully_connected_per_channel_s8`. CMSIS-NN takes int32 arrays, so tensors avoid any runtime conversion. `int[]` lists would also cost about 20 bytes per output feature in the program, where a tensor costs 4. The two C++ entry points share one implementation that takes the CMSIS-NN call; `quantized_linear` is unchanged. The Python reference requantizes each output feature with its own parameters. This diff was authored with Claude Code. Differential Revision: D126929702

AdrianLundell

Looks good overall. a few small nits. For easier reviewing I would appreciate to keep different PR:s separate, otherwise I have to sort out non relevant changes myself

AdrianLundell · 2026-11-04T13:25:12Z

+namespace {
+// Runs `fully_connected`, a CMSIS-NN fully connected call bound to its
+// requantization parameters, on the op's tensors.
+template <typename FullyConnected>


Templating this adds too much complexity for too little benefit for my taste.

AdrianLundell · 2026-11-04T13:26:58Z

+    else:
+        output = torch.stack(
+            [
+                requantize_cmsis(output_reshaped[..., channel], int(mult), int(shift))


Maybe simpler to add per-channel support to requantize_cmsis?

AdrianLundell · 2026-11-04T13:28:29Z

        ]
        super().__init__(quantizers)

+    def set_node_target(


This is from another PR

rascani · 2026-11-04T16:13:15Z

Looks good overall. a few small nits. For easier reviewing I would appreciate to keep different PR:s separate, otherwise I have to sort out non relevant changes myself

This is happens when internal diff stacks are exported one by one. It needs the earlier diffs so there's a functional stack.

@pssrawat - when working with stacks internally, you can use ghexport to get a corresponding PR stack. I can provide instructions if you'd like to try that.

Summary: `cortex_m::quantized_linear` supports only per-tensor weights: it calls `arm_fully_connected_s8` with a single requantization multiplier and shift. This adds per-channel linear weights. With `CortexMQuantizer.set_node_target` (previous diff), `set_node_target(torch.ops.aten.linear.default, INT8_PER_CHANNEL_CONFIG)` gives per-channel int8 linear weights; without it, linear layers keep `per_tensor_config`, so existing exports are unchanged. `CortexMLinearCheck` rejects per-channel weights that are not quantized per output feature (axis 0, or unset as the convolution check allows), and the lowering checks that it gets one weight scale per output feature, since the kernel indexes its multipliers by output feature. Per-channel linear weights lower to a new `cortex_m::quantized_linear_per_channel`. It matches `quantized_linear` but takes int32 multiplier and shift tensors, as the convolution ops do, and calls `arm_fully_connected_per_channel_s8`. CMSIS-NN takes int32 arrays, so tensors avoid any runtime conversion. `int[]` lists would also cost about 20 bytes per output feature in the program, where a tensor costs 4. The two C++ entry points share one implementation that takes the CMSIS-NN call; `quantized_linear` is unchanged. The Python reference requantizes each output feature with its own parameters. This diff was authored with Claude Code. Differential Revision: D126929702

pssrawat · 2026-11-04T23:41:17Z

This is happens when internal diff stacks are exported one by one. It needs the earlier diffs so there's a functional stack.

@pssrawat - when working with stacks internally, you can use ghexport to get a corresponding PR stack. I can provide instructions if you'd like to try that.

Addressed the nits. The changes from bottom-stack no longer present.

AdrianLundell · 2026-11-05T15:20:47Z

  cmsis_nn_dims output_dims = {batches, 1, 1, out_feat};

-  arm_cmsis_nn_status status = arm_fully_connected_s8(
+  const arm_cmsis_nn_status status = arm_fully_connected_wrapper_s8(


I think that this might link both per_channel and per_tensor kernels even if just one is used, could you verify if this is the case and fix that in that case?

Good catch, confirmed. arm_fully_connected_wrapper_s8 references both arm_fully_connected_s8 and arm_fully_connected_per_channel_s8. Even with --gc-sections, a binary that only uses per-tensor linear still linked the per-channel kernel.
I've removed the wrapper. quantized_linear_impl now takes the kernel as a template argument, and each op passes its own kernel.

Summary: `cortex_m::quantized_linear` supports only per-tensor weights: it calls `arm_fully_connected_s8` with a single requantization multiplier and shift. This adds per-channel linear weights. With `CortexMQuantizer.set_node_target` (previous diff), `set_node_target(torch.ops.aten.linear.default, INT8_PER_CHANNEL_CONFIG)` gives per-channel int8 linear weights; without it, linear layers keep `per_tensor_config`, so existing exports are unchanged. `CortexMLinearCheck` rejects per-channel weights that are not quantized per output feature (axis 0, or unset as the convolution check allows), and the lowering checks that it gets one weight scale per output feature, since the kernel indexes its multipliers by output feature. Per-channel linear weights lower to a new `cortex_m::quantized_linear_per_channel`. It matches `quantized_linear` but takes int32 multiplier and shift tensors, as the convolution ops do, and calls `arm_fully_connected_per_channel_s8`. CMSIS-NN takes int32 arrays, so tensors avoid any runtime conversion. `int[]` lists would also cost about 20 bytes per output feature in the program, where a tensor costs 4. The two C++ entry points share one implementation that takes the CMSIS-NN call; `quantized_linear` is unchanged. The Python reference requantizes each output feature with its own parameters. This diff was authored with Claude Code. Differential Revision: D126929702

AdrianLundell · 2026-11-09T15:48:19Z

-// cppcheck-suppress unusedFunction
-Tensor& quantized_linear_out(
+namespace {
+template <typename QuantParams, typename FullyConnectedFn>


For me I would prefer just copying the function twice over templating it but this is fine too.

rascani

Review automatically exported from Phabricator review in Meta.

pssrawat requested a review from rascani as a code owner November 1, 2026 04:00

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 1, 2026

meta-codesync Bot added the meta-exported label Nov 1, 2026

meta-codesync Bot changed the title ~~Per-channel int8 quantized linear~~ Per-channel int8 quantized linear (#24397) Nov 2, 2026

pssrawat force-pushed the export-D126929702 branch from 47819b6 to 63e5911 Compare November 2, 2026 19:53

pssrawat added the release notes: none Do not include this in the release notes label Nov 2, 2026

pssrawat requested a review from AdrianLundell November 2, 2026 22:21

pssrawat force-pushed the export-D126929702 branch from 63e5911 to 73f94ec Compare November 3, 2026 04:12

AdrianLundell requested changes Nov 4, 2026

View reviewed changes

pssrawat force-pushed the export-D126929702 branch from 73f94ec to bd55896 Compare November 4, 2026 21:48

pssrawat force-pushed the export-D126929702 branch from bd55896 to ffc2a1a Compare November 4, 2026 23:24

pssrawat force-pushed the export-D126929702 branch from ffc2a1a to 3fc45c1 Compare November 4, 2026 23:38

pssrawat requested a review from AdrianLundell November 5, 2026 05:07

AdrianLundell reviewed Nov 5, 2026

View reviewed changes

pssrawat force-pushed the export-D126929702 branch from 3fc45c1 to 499a468 Compare November 5, 2026 17:24

pssrawat requested a review from AdrianLundell November 5, 2026 17:26

AdrianLundell approved these changes Nov 9, 2026

View reviewed changes

rascani approved these changes Nov 9, 2026

View reviewed changes

Merge branch 'main' into export-D126929702

bd2a5a5

meta-codesync Bot merged commit a168321 into pytorch:main Nov 10, 2026
204 checks passed

Conversation

pssrawat commented Nov 1, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24397

✅ No Failures

Uh oh!

meta-codesync Bot commented Nov 1, 2026

Uh oh!

AdrianLundell left a comment

Choose a reason for hiding this comment

Uh oh!

AdrianLundell Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

pssrawat Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

AdrianLundell Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

AdrianLundell Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

rascani commented Nov 4, 2026

Uh oh!

pssrawat commented Nov 4, 2026

Uh oh!

AdrianLundell Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

pssrawat Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

AdrianLundell Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

rascani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pssrawat commented Nov 1, 2026 •

edited by meta-codesync Bot

Loading

pytorch-bot Bot commented Nov 1, 2026 •

edited

Loading