No integrity verification for cached models: a cache with wrong bytes reports healthy at every step and fails only at `LoadAsync`, with QNN 14001

[aclinick]

Summary

A cached model can end up on disk at the correct size with the wrong bytes. Nothing in
the SDK notices. IsCachedAsync
returns true, GetPathAsync returns a populated directory, the reported size and
revision match, and a subsequent DownloadAsync returns immediately because the model
is considered present. The only symptom appears much later, at LoadAsync, as an
opaque QNN error 14001.

The result is that an application has no supported way to tell a corrupt cache apart
from an unsupported machine, and no supported way to repair one. Both present
identically: the model is cached, and it will not load.

Environment

SDK	Microsoft.AI.Foundry.Local.WinML 1.0.0 (Foundry Local Core 1.1.0)
Model	qwen2.5-7b-instruct-qnn-npu:3 (alias qwen2.5-7b)
ORT	1.23.x native bundled with the SDK, Microsoft.ML.OnnxRuntime.Managed 1.23.2
Host	Snapdragon X Elite (X1P64100), Windows 11 arm64 10.0.26200
EP	MicrosoftCorporationII.WinML.Qualcomm.QNN.EP.1.8 1.8.30.0, QnnHtp 2.40.0.0
NPU driver	30.0.220.3000, Compute DSP 2.0.4478.2200
App	MSIX-packaged WinUI 3 / .NET 10 desktop app

Two physically identical machines with byte-identical EP, driver, SoC, OS build and
RAM. The model loads on one and fails on the other.

What we observed

On the failing machine, LoadAsync throws with ONNX Runtime text forwarded verbatim:

Failed to create context from binary
  QnnBackendManager::LoadCachedQnnContextFromBuffer
  Error code: 14001

Before reaching a cache theory we eliminated, on both machines:

• Execution provider. EP.1.8 1.8.30.0 is self-contained: it ships QnnHtp.dll,
  QnnHtpV73Stub.dll, libQnnHtpV73Skel.so and libqnnhtpv73.cat. Nothing is missing
  on the failing box.
• Driver and SoC. Identical, to the build number.
• Signature catalogs. No skel catalogs registered in CatRoot on either machine.
• Model size and memory. qwen2.5-0.5b-instruct-qnn-npu,
  qwen2.5-1.5b-instruct-qnn-npu and phi-3.5-mini-instruct-qnn-npu all load
  correctly on the failing machine, and all three are built with the same QNN SDK
  2.37.1 as the 7B. So the EP, the driver, the SDK version and the NPU itself are all
  working on that machine.

That left the bytes. We captured a manifest of the model cache from the working
machine (33 files, 6.81 GB, SHA-256 per file) and compared it against the failing
machine. Exactly one file differed:

CORRUPT   context_1_ctx_qnn.bin, right size but different contents
          470,421,504 bytes on both machines

Deleting the cached model and downloading it again fixed the machine.

We had watched Foundry's downloader abort and resume twice at roughly 59% earlier in
the same session, with 116 GB free on the target volume. That is the most likely
mechanism: a resumed range write that lands the file at full length with a bad
interior region. We want to be clear that this is inference and not something we
proved, which is why the report above is framed around detection rather than cause.

These bytes were not written by our code. The application never fetches model
files itself. Every download goes through ModelVariant.DownloadAsync with a progress
callback and nothing else; the only HttpClient in the codebase talks to an unrelated
local service. We also apply no timeout to the download, so we never cancel one on a
timer. The one way our process could interrupt a transfer is the user closing the app
mid-download, which is an ordinary real-world interruption rather than an induced one,
and is precisely the case a resume path needs to survive.

Why this is hard to handle from an application

The failure surfaces roughly 470 MB and one process boundary away from its cause, and
every signal the SDK offers says the cache is fine:

1. IsCachedAsync returns true. It appears to check for presence, not integrity.
   There is no checksum verification at the end of a download and no way to request one.
2. DownloadAsync is a no-op once cached. An app that catches the load failure and
   retries the download gets an immediate success and the same broken bytes. There is
   no force or verify parameter.
3. There is no obvious supported repair. RemoveFromCacheAsync exists but is not
   discoverable as the documented remedy for this, and nothing in the load exception
   suggests reaching for it.
4. The error text names no file. QNN 14001 identifies neither the context binary
   nor the cache path, so the natural reading is "this machine cannot run this model",
   which is exactly the wrong conclusion and sends you off measuring RAM and driver
   versions, as it sent us.

To ship, we had to build all of this ourselves: capture a reference manifest on a known
good machine, ship it as an app asset, hash 6.81 GB of cache on the user's machine, and
add our own delete-and-redownload path. That is a lot of machinery to compensate for a
download that did not verify itself, and it only works because we happened to have a
second machine to take a reference from. An application shipping to real users has no
such machine.

Steps to reproduce

The corruption itself depends on an interrupted transfer, so the deterministic
reproduction is to simulate the result rather than the cause. This demonstrates the
detection gap, which is the part we are asking about:

1. Download a QNN NPU model, for example qwen2.5-7b-instruct-qnn-npu:3, and confirm
   it loads.

2. Locate the cache via GetPathAsync. In our case
   %USERPROFILE%\.<AppName>\cache\models\Microsoft\qwen2.5-7b-instruct-qnn-npu-3\v3.

3. Corrupt a context binary in place without changing its length:

   $f = "<cache>\context_1_ctx_qnn.bin"
   $fs = [IO.File]::Open($f, 'Open', 'Write')
   $fs.Seek(250MB, 'Begin') | Out-Null
   $fs.Write((New-Object byte[] 4096), 0, 4096)
   $fs.Dispose()

4. In a fresh process:

   var variant = await catalog.GetModelVariantAsync("qwen2.5-7b-instruct-qnn-npu:3");
   Console.WriteLine(await variant.IsCachedAsync());  // true
   await variant.DownloadAsync();                     // returns immediately, no repair
   Console.WriteLine(await variant.IsCachedAsync());  // still true
   await variant.LoadAsync();                         // throws, QNN 14001

Expected: either the download verifies and repairs the file, or IsCachedAsync
reports the cache as unusable, or the load error identifies the file that failed to
parse.

Actual: the cache reports healthy at every step and the load fails with an error
that points at the accelerator rather than at the bytes.

Suggested fixes, roughly in order of value

1. Verify after download. Compare each file against the size and hash the service
   already publishes in the model manifest, and fail the download rather than the load.
   This alone would have turned a multi-day investigation into an error message.
2. Make DownloadAsync repairable. A force: true or verify: true option so an
   application can recover without reverse-engineering the cache layout.
3. Add an integrity API. Something like VerifyCacheAsync returning the offending
   files, so applications do not each have to reimplement hashing and ship their own
   reference manifests.
4. Name the file in the load failure. Wrapping ORT's message with the context
   binary path and the cache directory would let a developer reach the answer directly
   from the exception.
5. Harden the resume path. If a resumed transfer cannot prove it continued from the
   right offset, restart it instead of stitching.

Item 1 is the one that matters. The rest are mitigations for the case where it fails
anyway.

Related

The same investigation ran into a second, separate problem worth mentioning because it
compounds this one: FoundryLocalManager.IsInitialized flips to true after
CreateAsync but before DownloadAndRegisterEpsAsync completes. Touching the
catalog in that window caches CPU-only variants for the lifetime of the process, so an
NPU model silently never gets offered. We work around it by awaiting EP registration
explicitly before any catalog access. Happy to file that separately if useful.