Feature Request: Is it possible to bring back this CUDA backend only feature from Pull Request #13529

[InfernalDread]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hello,

I was very excited about this pull request: #13529

URL Link: #13529

Even if it is only for CUDA, if it is possible to maintain this without major issues, it would be a massive quality of life improvement for those that do not have beefy hardware.

Thank you for your time and consideration.

Motivation

This will allow those without great hardware to experience greater context lengths and better performance from MOE models by being able to offload more layers onto GPU due to the decrease of memory requirement from KV Cache.

Possible Implementation

No response

[JohannesGaessler]

As implied in the PR, a CPU implementation is needed.

[InfernalDread]

Ah, so its impossible to get the CUDA to work without CPU implementation. Thanks for the quick reply!