Skip to content

Arm backend: Enable VGF KV-cache aliasing in LLM export - #24680

Merged
xingguo01 merged 2 commits into
pytorch:mainfrom
xingguo01:arm-backend-vgf-kv-cache-export-aliasing
Nov 9, 2026
Merged

xingguo01 merged 2 commits into
pytorch:mainfrom
xingguo01:arm-backend-vgf-kv-cache-export-aliasing

Conversation

@xingguo01

@xingguo01 xingguo01 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Expose mutable buffer aliasing through the LLM export configuration and forward it to the VGF partitioner for static quantized KV caches.

Update the SmolLM2 VGF example with the supported export command, persistent runner handling, timestamp reporting, and parallel WikiText perplexity scoring. Wait for runner completion before reading logits, kill hung runners during cleanup, and let evaluation select the KV input dtype. Support tokenizers without a BOS token.

AI-assisted-by: Codex

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

@pytorch-bot

pytorch-bot Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24680

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Pending, 1 Unclassified Failure

As of commit f7481c6 with merge base 3d5172b (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 5, 2026
@github-actions github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Nov 5, 2026
@xingguo01 xingguo01 added the release notes: arm Changes to the ARM backend delegate label Nov 5, 2026
Expose mutable buffer aliasing through the LLM export configuration and
forward it to the VGF partitioner for static quantized KV caches.

Update the SmolLM2 VGF example with the supported export command,
persistent runner handling, timestamp reporting, and parallel WikiText
perplexity scoring. Wait for runner completion before reading logits,
kill hung runners during cleanup, and let evaluation select the KV
input dtype. Support tokenizers without a BOS token.

AI-assisted-by: Codex
Change-Id: Id001b3fde168262ed748b03498d278d594bbfa0b
Signed-off-by: Xingguo Li <xingguo.li@arm.com>
Read mutated non-persistent buffers from ExportedProgram.constants
when resetting static VGF KV state. Keep state_dict lookup for
persistent buffers so both storage paths match the graph signature.

Cover both persistence modes with StaticQuantizedKVCache, including
zero initialization, retained scales, subsequent updates, and dtype
validation.

AI-assisted-by: Codex
Change-Id: Ie8364ccb6f9ddb3ea680e2aaf8b71508a564e699
Signed-off-by: Xingguo Li <xingguo.li@arm.com>
@xingguo01
xingguo01 force-pushed the arm-backend-vgf-kv-cache-export-aliasing branch from 3330024 to f7481c6 Compare November 9, 2026 15:17
@xingguo01
xingguo01 merged commit 4411b8d into pytorch:main Nov 9, 2026
488 of 490 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend release notes: arm Changes to the ARM backend delegate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants