Feature Request: Upstream the TurboQuant implementation

[aminya]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Is there any plan to upstream the changes? The implementation has been very stable based on my tests for the past several weeks. I believe, it's time to upstream the changes.

Motivation

Upstreaming the changes means having TurboQuant on the main release channel, and more velocity for improvements. Iterations can be done later if needed.
Right now, the users of this fork are missing out on thr latest improvements from the upstream and vice versa.

Possible Implementation

TheTom/turboquant_plus/issues/27

[theoparis]

I noticed that upstream has fixed compilation with openssl 4.0.0, while this fork lacks the changes to the vendored cpp-httplib to solve it. So upstreaming this would be great!

[TheTom]

Yes, upstreaming is the goal. Tracked at TheTom/turboquant_plus#27.

Two real constraints on how fast this can move.

Constraint 1: maintainer caution on TurboQuant work specifically. The TurboQuant discussion at ggml-org#20969 is at 544 comments. There have been 10+ TurboQuant-related PRs from various contributors over the past two months. Most got closed without merge. Two were closed with the explicit "(AI policy violation)" tag (ggml-org#21307, ggml-org#21818), which tells you exactly how careful the maintainers are being about what TQ work lands in tree. CISC has pointed people at the CONTRIBUTING.md "adding new data types" section more than once. This is the right posture for a new quant type, not a complaint, but it sets the pace.

Constraint 2: my own in-flight PRs are not moving. Two open right now:

• metal : add GATED_LINEAR_ATTN op ggml-org/llama.cpp#21452 (Metal GATED_LINEAR_ATTN op), open since Apr 5
• cuda: LRU eviction + overalloc for legacy pool ggml-org/llama.cpp#22207 (CUDA LRU eviction + overalloc), open since Apr 21

These are the small, individually-justifiable pieces that have to land before a full TurboQuant+ PRs are realistic. They are sitting in review queue. I cannot ship a "here is the new turbo3/turbo4 KV cache type" PR ahead of these without the review burden becoming unmanageable. This is part of their first time contribution guidelines. Have to get in some basic PRs first to prove yourself.

Near term I can cherry-pick or sync important upstream fixes into this fork. The openssl 4.0.0 thing @theoparis flagged is a good example, will pull that in. Long term the upstream PR is the right destination so the fork stops accumulating drift.

TL;DR: I want this upstream as much as you do. The bottleneck is not my willingness, it is the rate at which the llama.cpp maintainers can review TurboQuant work given the volume of low-quality submissions they have already had to triage. Will keep this issue updated as the upstream PR progresses.

[TheTom]

#121 for openssl cherry-picks

[aminya]

@TheTom the upstream has merged MTP implementation in its entirety. I believe it'd be possible to make a single TurboQuant PR for upstream to encourage them with the reviews. The one off PRs are now ignored under the noise of big features.
ggml-org#22673

[apollo-mg]

I merged a (can't remember which) MTP PR into this fork on my build without issue. Worked pretty well. Definitely faster, uses a pretty big chunk of VRAM to enable it.

[TheTom]

Im currently working on updating my fork. I still have outstanding PRs for upstream and am not allowed to make more without upsetting the maintainers...

Apple was certainly faster, they took my swift CLs pretty quickly.

[apollo-mg]

Im currently working on updating my fork. I still have outstanding PRs for upstream and am not allowed to make more without upsetting the maintainers...

Apple was certainly faster, they took my swift CLs pretty quickly.

I've seen how they treat your PR's over there...

[TheTom]

#146

Please test this upstream sync