MTP & Desktop Support + More

[hg0428]

In the Readme you said you are mainly only targeting iOS/Android.
What about MacOS, Windows, and Linux?
I am attempting to make a single chat app that can work across platforms including both mobile and desktop, so having support for all platforms is a key part of that.

Additionally I would love native builtin support for embedding models here as well, a key part of various RAG pipelines.
Also I would like to see support for MTP, which was recently added to Llama.cpp. This can help increase speeds with speculative decoding which could be of great help on mobile.

And another thing that I would really love to have native support for is batching multiple sequences at once for higher parallelism, as well as potentially (though less urgent) server-style continuous batching.

I believe most of these things are supported within Llama.cpp itself and just need to be exposed and optimized for this library.

[netdur]

Thanks @hg0428, these are reasonable asks, and most of them do map onto things llama.cpp already supports

MTP yes, it landed in llama.cpp recently and I'm planning to wire it into the binding

Embeddings good catch, this was supported before the 0.9 rewrite and got dropped during it, that was an oversight, not a scope decision, I'll add it back

Platform support this is really a "who builds the native binaries" question, not a binding limitation, the Dart/FFI layer is hardware-agnostic, it loads whatever native lib you point it at, what I ship from CI:

• iOS / macOS: Metal, simple story, xcframework + dylib already provided
• Android: CPU and Hexagon (Snapdragon) AARs but that doesn't cover the full Android GPU/NPU landscape
• Linux: the hardware matrix is too wide for me to build and maintain as a solo project

So for Linux/Windows desktop: if you build the native binaries for your specific target hardware, the binding will work — there's nothing platform-specific in the Dart layer stopping it, I just can't commit to producing or maintaining those builds myself

Batching / continuous batching continuous batching is something I'm considering bringing back, but it's not in current plans, for heavier server-style / multi-process workloads I split that role out into a separate Rust project (hugind), which may be relevant for the desktop side of a cross-platform app

I have no idea what is "batching multiple sequences", can you explain it?

[hg0428]

Thanks @hg0428, these are reasonable asks, and most of them do map onto things llama.cpp already supports

MTP yes, it landed in llama.cpp recently and I'm planning to wire it into the binding

Embeddings good catch, this was supported before the 0.9 rewrite and got dropped during it, that was an oversight, not a scope decision, I'll add it back

Platform support this is really a "who builds the native binaries" question, not a binding limitation, the Dart/FFI layer is hardware-agnostic, it loads whatever native lib you point it at, what I ship from CI:

• iOS / macOS: Metal, simple story, xcframework + dylib already provided
• Android: CPU and Hexagon (Snapdragon) AARs but that doesn't cover the full Android GPU/NPU landscape
• Linux: the hardware matrix is too wide for me to build and maintain as a solo project

So for Linux/Windows desktop: if you build the native binaries for your specific target hardware, the binding will work — there's nothing platform-specific in the Dart layer stopping it, I just can't commit to producing or maintaining those builds myself

Batching / continuous batching continuous batching is something I'm considering bringing back, but it's not in current plans, for heavier server-style / multi-process workloads I split that role out into a separate Rust project (hugind), which may be relevant for the desktop side of a cross-platform app

I have no idea what is "batching multiple sequences", can you explain it?

Hi, thanks for the information.

As for batching multiple sequences, what I meant was running multiple sequences at once in parallel.

[netdur]

@hg0428 the MTP landed, I finally had time to test it, but check README notes about it