[WIP]backend: Integrating QNN (Qualcomm AI Engine Direct) as a dedicated backend for Qualcomm NPUs

[chraac]

Warning: This is an early draft of my fork and will continue to be updated to meet the requirements in the contributing guidelines

Summary

This fork is based on zhouwg's initial PR and performs further refactoring and improvements to introduce support for the Qualcomm QNN backend to GGML.

This backend is organized into three distinct integration layers:

graph TB
    subgraph GGML Adaptation Layer
        A1[Graph Caching, Mapping, and Execution]
        A2[Tensor Binding and Execution Flow]
    end

    subgraph QNN Object Layer
        B1[QNN System and Instance Management]
        B2[Dynamic Resource Handling]
    end

    subgraph Utility Layer
        C1[Dynamic Library Loading & Search Path Management]
        C2[General Utilities]
    end

    %% Relations to illustrate stack dependency
    A1 -->|Uses| B1
    A2 -->|Uses| B1
    B1 -->|Relies on| C1

Loading

1. GGML Adaptation Layer

   • Graph Caching, Mapping, and Execution:

     • Provides a robust mechanism to map a GGML computation graph into a corresponding QNN graph, allowing efficient offloading of operations to the QNN accelerator.
     • Implements graph caching strategies (in backend-ops.cpp) to minimize redundant graph creation and boost execution performance.
     • Seamlessly translates GGML operations into corresponding QNN op objects using specialized op constructors and configuration functions (configured in op-config-caps.cpp and op-config-impl.cpp).

   • Tensor Binding and Execution Flow:

     • Adapts GGML tensor objects to the QNN backend (see tensor.hpp and graph.hpp), managing both host and RPC memory via buffer interfaces like qnn_buffer_interface.
     • Ensures proper data flow between GGML graphs and QNN execution contexts through carefully handled tensor binding/unbinding procedures.

2. QNN Object Layer

   • QNN System and Instance Management:

     • Encapsulates the QNN system via the qnn_system_interface class, originally derived from executorch, to create and free the QNN system context.
     • Manages QNN instance creation and initialization via the qnn_instance class
     • Implements backend loading routines (e.g., load_backend() and load_system()) that retrieve provider lists and choose valid QNN interfaces based on API version checks.
     • Uses caching mechanisms for loaded backends and tracks library handles to guarantee proper cleanup during finalization.

   • Dynamic Resource Handling:

     • Integrates fallback mechanisms in load_lib_with_fallback() to reliably load both the system and RPC libraries.
     • Manages RPC memory allocation and deallocation via function pointer resolution from the loaded RPC library.

3. Utility Layer

   • Dynamic Library Loading & Search Path Management:

     • Implements functions in qnn-lib.cpp to manage dynamic library loading with fallbacks.
     • Uses helper routines such as insert_path() and set_qnn_lib_search_path() to configure environment variables (like LD_LIBRARY_PATH on Linux and ADSP_LIBRARY_PATH on Android) based on a custom library search path.

   • General Utilities:

     • Provides detailed error and debug logging through QNN logging macros.

Key Features and Improvements

• Graph Mapping Mechanism:

  • Efficient mapping of GGML graphs into QNN graphs is a standout feature, enabling the offloading and execution of computation graphs on hardware accelerators (see graph.hpp and backend-ops.cpp).
  • Graph caching strategies help reuse QNN graphs to reduce redundancy and enhance performance.
  • The translation of GGML operations into corresponding QNN ops supports various data types and parameter configurations.

• Backend Context and Device Management:

  • Comprehensive QNN instance initialization supports API negotiation, enhanced error handling, and detailed device property logging.
  • Detailed logs (chipset description, HTP architecture, VTCM memory size) facilitate debugging and performance tuning.

Build

For build instructions please refer to this page

Testing

• Basic functionality of the QNN backend has been verified on Android, Linux, and Windows platforms using test-backend-ops—this is integrated into the pipeline for each commit node of the dev-refactoring branch.

  Platform	test-backend-ops	full console output
  Android		test-backend-ops_all_android_ff033e1.log
  Linux		test-backend-ops_all_linux_ff033e1.log
  Windows	To be fill

• Proper graph creation and execution paths are confirmed through detailed log messages.

• Memory registration and cleanup within tensor binding functions have been thoroughly checked.

• Table below shows GIFs of qnn backend running on different platforms

  Platform	Soc	Model	Gif	Origin video
  Android	8 Gen 2	llama-3-8B-Instruct-Q4_K_M		Recording_Muted_hevc.mp4
  Windows	To be fill

Current state

• The test-backend-ops suite passes on all platforms, including support for both qnn-npu and qnn-gpu devices.
• Testing with llama3.2-1b/3b-f16/32 models yields expected results.
• Quantized matrix multiplication is under development; for quantized modules, the CPU backend may be used as a fallback.

Future development

• Further feature support and device-specific optimizations are planned (see also the project backlog).
• Future iterations will add support for quantization data types, with efforts underway to map GGML's block quantization structure into QNN.

[chraac]

I don't know this Chinese programmer and I'm not a member of his team and I'd like to see his team's success in this great community. thanks.

Yeah, just to clarify, @zhouwg is not affiliated with us, but we appreciate his support! Anyone interested in discussing QNN-related topics is very welcome to join the conversation.

[chraac]

here's how we map ggml_cgraph into a qnn graph

[chraac]

TODO: this dl_loader can be remove if upstream provide a unified dynamic load machanism

llama.cpp/ggml/src/ggml-backend-reg.cpp

Line 99 in 34a846b

static dl_handle * dl_load_library(const std::wstring & path) {

[chraac]

I didn't provide any support to @chraac and his team. as I said before: I don't know this guy and his team and I'd like to see their success in this community. thanks so much.

I'd like to rephrase my previous statement. I appreciate your earlier work, as my fork is based on your initial PR

[oreomaker]

Great effort! According to QNN Shared Memory Doc, the the _rpc_buffer in HTP can be directly accessed by CPU. Maybe there can be a no copy implementation.

[chraac]

Yeah, thank you for the reminder! current the rpc buffer is disabled:

    bool should_use_mem_handle() const {
        // TODO: figure out how to set rpc mem to multiple tensor
        return false;
    }

thought we can reuse the rpc buffer for backing ggml tensor in the future, but now its disable by default

[chraac]

have an item in my project backlog here: qnn backend (view)

[chraac]

here's how we create corresponding mat_mul op, and the op will looks like:

which following ggml's guide line:
https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md

[chraac]

Generates a unique key for a given ggml_cgraph. The key is constructed by concatenating the descriptions of the operations and their associated tensor dimensions within the graph.

Example key format: MUL_MATf32_256x16x10f32_256x1x10f32#LOG#ADD#ADDf32_16x1x10f32

May need some refactoring here to handle more complex graph structures and edge cases

[chraac]

I never drop such a comment in other's PR, this is my first time in this great tech community which is out of mainland China,sorry to waste resource and time in public community, thanks.

Notice you've edited your original post with additional information. I'd like to clarify that my intent was to address specific technical issues that have existed throughout your PR series. without implementing correct matrix transposition, the mul_mat operation cannot function properly.

And to reiterate: please focus on improving your codebase in an objective manner without making assumptions about or judging others' work.

If you have any thoughts on my source code implementation, would be very welcome! I'm open to discussion about the design, implementation details, or any other technical aspects of the code.

Collaborative feedback helps us all build better software. By sharing insights about implementation approaches, performance considerations, and edge cases, we collectively create more reliable and efficient code than any individual contributor could achieve independently. (Not gonna lie - it can be tough sometimes, but I'm all about keeping an open mind and hearing different viewpoints. Just trying my best here!)

[chraac]

In our recent PR, we added a counter to track which operations are successfully offloaded to the qnn backend. While testing with the llama-3-8B-Instruct-Q4_K_M model, found an interesting result:

Current Status

• Even though quantized tensor support isn't implemented yet, many operations are still being processed by the qnn backend since they operate on F32 data
• As shown in the screenshot, we're seeing significant operation offloading opportunities
• However, no MUL_MAT op are currently being offloaded to qnn, which are critical for performance

Next Steps

Based on this analysis, I'm shifting focus a bit to implement support for additional operation types that can be offloaded from cpu to qnn - this will provide immediate performance benefits while running models on device...
Simultaneously, will continue investigating how to port GGML's quantization scheme to QNN - this remains a core objective for our long-term performance goals, especially for quantized models like the one shown in the testing.

Test method and Resources

1. Push llm model to android device folder /data/local/tmp
2. Run scripts/run_device_model.sh --verbose --model-name 'meta-llama_Meta-Llama-3-8B-Instruct-Q4_K_M.gguf', run_device_model.sh can be found here

Full running log:
run_model.8b.q4.debug.log

[conradev]

@chraac Does this backend work simultaneously with the Adreno OpenCL backend?

Is the idea to offload as much as possible to the NPU, then OpenCL, and then the CPU?

[chraac]

Does this backend work simultaneously with the Adreno OpenCL backend?

Hi @conradev,
Thank you for sharing that link. while both solutions aim to improve performance on Qualcomm SoCs, they take different approaches:

• the Adreno OpenCL backend is specifically optimized for Adreno GPUs, building on the original OpenCL backend with Adreno-specific optimizations.

• my implementation leverages QNN SDK, which is Qualcomm's official ML inference framework. it works as a higher-level abstraction layer that maps GGML operations to QNN's native operations. This approach can target multiple compute devices (CPU, GPU, and NPU) on Qualcomm platforms, providing greater flexibility in deployment scenarios.

these implementations represent two distinct but complementary approaches to hardware acceleration on Qualcomm devices - one focused specifically on Adreno GPU optimization via OpenCL, and the other providing a vendor-supported framework integration with broader device support.

Is the idea to offload as much as possible to the NPU, then OpenCL, and then the CPU?

Short answer: It's up to the GGML framework's scheduler to make that decision.

In our implementation, we simply provide capability information to GGML about whether each QNN device (CPU/GPU/NPU) can support specific operations. We also indicate the device type (CPU/GPU/ACCEL) for each QNN backend. The GGML framework then uses this information to determine which device should execute each operation.

For the llama-3-8B-Instruct-Q4_K_M , we've observed that the scheduler tends to prefer qnn-gpu over qnn-npu for many operations. This preference is likely based on the device type classifications we provided to the scheduler.

[chraac]

I'm considering set all QNN devices (CPU/GPU/NPU) as GGML_BACKEND_DEVICE_TYPE_ACCEL. this approach would give them the same scheduling priority during offloading and should result in more operations being scheduled to the NPU.
@conradev , I'd appreciate your thoughts on this approach. thanks!

[chraac]

I already blocked in this community before 02/16/2025 because of my stupid mistake last year which part of reasons came from this CN programmer in my first PR and which the main reason is my personal mistake, this CN programmer has already intended to use the maintainers 's hands to block me again in my third PR so his voice and misinformation can be seen by everyone in this tech community.

Let's see what @slaren said in you PR:

Hi @zhouwg. I want to clarify that the comments made by @chraac in your previous PR had no influence whatsoever in the decision to block you from participating in this repository. Technical feedback and code reviews are always welcome and even encouraged. However, you were blocked due to a consistent pattern of comments that incited personal conflict, often in response to legitimate technical feedback. The comments linked by @chraac (now removed) are an example of this behavior.

I'm focused on improving the QNN backend support and welcome technical discussions on this topic. As the maintainer noted, provoking personal conflict isn't encouraged. Comments that stray from technical feedback will not receive a response from now on.

[chraac]

Hi @slaren, noticed we have similar dynamic library loading functionality in ggml-bacnend-reg.cpp (the dl_load_library function) that could be useful in other parts of the codebase.
I suggest moving this to a common utility module so we can reuse it across the project. This would help reduce code duplication and provide a consistent approach to loading libraries.
I'd be happy to prepare another PR about that, WDYT?

[slaren]

Sorry, I missed this. I think that this code is small enough that it is not really a problem if it is duplicated in a backend, and making it part of the public API available to backends may make it harder to change it in the future. So at the moment my preference would be to avoid this.