Sleeping mid-download leaves a full length, partly zero filled model that nothing detects: the cache reports healthy at every step and fails only at `LoadAsync`, with QNN 14001

[aclinick]

Summary

Sleeping the machine during a model download leaves the file on disk at its correct
final length with the undownloaded remainder reading as zeros, and nothing in the SDK
notices. IsCachedAsync
returns true, GetPathAsync returns a populated directory, the reported size and
revision match, and a subsequent DownloadAsync returns immediately because the model
is considered present. The only symptom appears much later, at LoadAsync, as an
opaque QNN error 14001.

The cause is that the destination file is preallocated to its full length and is not
marked sparse, so a partly complete download is indistinguishable by size from a
complete one, and no content hash is recorded or checked anywhere in the pipeline. We
captured the intermediate state on disk; see "Update: we caught the cause on disk"
below for the state file and the valid data length measurement.

The result is that an application has no supported way to tell a corrupt cache apart
from an unsupported machine, and no supported way to repair one. Both present
identically: the model is cached, and it will not load.

Environment

SDK	Microsoft.AI.Foundry.Local.WinML 1.0.0 (Foundry Local Core 1.1.0)
Model	qwen2.5-7b-instruct-qnn-npu:3 (alias qwen2.5-7b)
ORT	1.23.x native bundled with the SDK, Microsoft.ML.OnnxRuntime.Managed 1.23.2
Host	Snapdragon X Elite (X1P64100), Windows 11 arm64 10.0.26200
EP	MicrosoftCorporationII.WinML.Qualcomm.QNN.EP.1.8 1.8.30.0, QnnHtp 2.40.0.0
NPU driver	30.0.220.3000, Compute DSP 2.0.4478.2200
App	MSIX-packaged WinUI 3 / .NET 10 desktop app

Two physically identical machines with byte-identical EP, driver, SoC, OS build and
RAM. The model loads on one and fails on the other.

What we observed

On the failing machine, LoadAsync throws with ONNX Runtime text forwarded verbatim:

Failed to create context from binary
  QnnBackendManager::LoadCachedQnnContextFromBuffer
  Error code: 14001

Before reaching a cache theory we eliminated, on both machines:

• Execution provider. EP.1.8 1.8.30.0 is self-contained: it ships QnnHtp.dll,
  QnnHtpV73Stub.dll, libQnnHtpV73Skel.so and libqnnhtpv73.cat. Nothing is missing
  on the failing box.
• Driver and SoC. Identical, to the build number.
• Signature catalogs. No skel catalogs registered in CatRoot on either machine.
• Model size and memory. qwen2.5-0.5b-instruct-qnn-npu,
  qwen2.5-1.5b-instruct-qnn-npu and phi-3.5-mini-instruct-qnn-npu all load
  correctly on the failing machine, and all three are built with the same QNN SDK
  2.37.1 as the 7B. So the EP, the driver, the SDK version and the NPU itself are all
  working on that machine.

That left the bytes. We captured a manifest of the model cache from the working
machine (33 files, 6.81 GB, SHA-256 per file) and compared it against the failing
machine. Exactly one file differed:

CORRUPT   context_1_ctx_qnn.bin, right size but different contents
          470,421,504 bytes on both machines

Deleting the cached model and downloading it again fixed the machine.

We had watched Foundry's downloader abort and resume twice at roughly 59% earlier in
the same session, with 116 GB free on the target volume. At the time we framed this
report around detection rather than cause, because the mechanism was inference. We have
since reproduced the cause and captured the on-disk state; see "Update: we caught the
cause on disk" below. The detection gap is still the primary ask, because it is what
leaves an application unable to recover.

These bytes were not written by our code. The application never fetches model
files itself. Every download goes through ModelVariant.DownloadAsync with a progress
callback and nothing else; the only HttpClient in the codebase talks to an unrelated
local service. We also apply no timeout to the download, so we never cancel one on a
timer. The one way our process could interrupt a transfer is the user closing the app
mid-download, which is an ordinary real-world interruption rather than an induced one,
and is precisely the case a resume path needs to survive.

Why this is hard to handle from an application

The failure surfaces roughly 470 MB and one process boundary away from its cause, and
every signal the SDK offers says the cache is fine:

1. IsCachedAsync returns true. It appears to check for presence, not integrity.
   There is no checksum verification at the end of a download and no way to request one.
2. DownloadAsync is a no-op once cached. An app that catches the load failure and
   retries the download gets an immediate success and the same broken bytes. There is
   no force or verify parameter.
3. There is no obvious supported repair. RemoveFromCacheAsync exists but is not
   discoverable as the documented remedy for this, and nothing in the load exception
   suggests reaching for it.
4. The error text names no file. QNN 14001 identifies neither the context binary
   nor the cache path, so the natural reading is "this machine cannot run this model",
   which is exactly the wrong conclusion and sends you off measuring RAM and driver
   versions, as it sent us.

To ship, we had to build all of this ourselves: capture a reference manifest on a known
good machine, ship it as an app asset, hash 6.81 GB of cache on the user's machine, and
add our own delete-and-redownload path. That is a lot of machinery to compensate for a
download that did not verify itself, and it only works because we happened to have a
second machine to take a reference from. An application shipping to real users has no
such machine.

Update: we caught the cause on disk

After filing the first draft we hit this again, and this time captured the intermediate
state. The trigger is machine sleep during a large download, which for a 7 GB model
on a laptop is close to unavoidable. On wake the transfer stopped advancing, then the
progress reporting ended as though it had finished.

The download writes through a chunked, parallel path (DownloadBlobChunked ->
DownloadChunkWithRetry -> WriteAtOffsetAsync) with resume state persisted next to
the payload as <file>.download.state. Here is a real one, mid-transfer:

{
  "BlobSize": 470364160,
  "ChunkSize": 2097152,
  "TotalChunks": 225,
  "BitmapByteAlignedStart": 0,
  "HighestCompletedChunk": 88,
  "CompletedCount": 47,
  "TruncCompletionBitmap": "IMqH+NRP+wzXXqEB",
  "LastModified": "2026-07-25T21:19:51.2341093Z"
}

47 of 225 chunks were complete. The corresponding iterator_3_ctx_qnn.bin on disk:

Property	Value
File length	470,364,160
BlobSize	470,364,160 (identical)
Valid data length	224,624,640 (47.75%)
Sparse flag	not set

The destination file is preallocated to its full final length on the first write. It
is not marked sparse, so it consumes its full size on disk and every byte past the valid
data length reads back as a well formed run of zeros. A file that is 47% downloaded is
indistinguishable, by length, from a file that is complete. We confirmed by reading at
chunk offsets: data past the valid data length is all zero, and the tail is not
recoverable from the file alone.

That is the whole failure. Any completeness check that trusts length, and there is no
hash in the persisted state to check instead, will accept this file. It explains the
symptom in the original report exactly: the cache reports healthy at every step, and the
first thing that actually reads the bytes is QNN, which fails at 14001.

Two smaller observations from the same artifact, offered as detail rather than as
separate asks:

• The bitmap and the data are not consistent with each other. Chunks marked incomplete
  below the high water mark still contained data, so the state file lags the writes. That
  direction is safe, it just re-downloads, but it does show the state file and the
  payload are not committed together, and the opposite direction would not be safe.
• HighestCompletedChunk (88) is far ahead of CompletedCount (47), which is expected
  with parallel chunks, but it means the completed region is full of holes rather than a
  clean prefix. Resume correctness rests entirely on the bitmap surviving accurately.

We could not find any content verification in the download path. The persisted state
carries a size and a LastModified, but no ETag and no content hash, and there is no
post-download hash comparison, so nothing in the pipeline can distinguish a zero filled
region from real model weights.

Why this points at the transport

Sleep is a hostile case for a long lived HTTPS transfer: sockets are dead on wake, and
the failure is silent rather than an error. Browsers and the OS download stacks have
spent years on exactly this, and package delivery already solves it too. An MSIX is
content addressed by 64 KB block hashes in a signed AppxBlockMap.xml, so a resumed or
partial transfer cannot be mistaken for a complete one, and blocks the machine already
has are not fetched again. This is one of the reasons we are asking, in the companion
report, for models to be shippable inside the application package: it moves this problem
onto a transport that has already solved it, rather than solving it again in the SDK.

Steps to reproduce

The cause is now known: interrupt a large model download by sleeping the machine, wake
it, and let the transfer end. The resulting file is full length with a short valid data
length.

The deterministic reproduction below simulates the result rather than the cause, and
still demonstrates the detection gap, which is the part we are asking about:

1. Download a QNN NPU model, for example qwen2.5-7b-instruct-qnn-npu:3, and confirm
   it loads.

2. Locate the cache via GetPathAsync. In our case
   %USERPROFILE%\.<AppName>\cache\models\Microsoft\qwen2.5-7b-instruct-qnn-npu-3\v3.

3. Corrupt a context binary in place without changing its length:

   $f = "<cache>\context_1_ctx_qnn.bin"
   $fs = [IO.File]::Open($f, 'Open', 'Write')
   $fs.Seek(250MB, 'Begin') | Out-Null
   $fs.Write((New-Object byte[] 4096), 0, 4096)
   $fs.Dispose()

4. In a fresh process:

   var variant = await catalog.GetModelVariantAsync("qwen2.5-7b-instruct-qnn-npu:3");
   Console.WriteLine(await variant.IsCachedAsync());  // true
   await variant.DownloadAsync();                     // returns immediately, no repair
   Console.WriteLine(await variant.IsCachedAsync());  // still true
   await variant.LoadAsync();                         // throws, QNN 14001

Expected: either the download verifies and repairs the file, or IsCachedAsync
reports the cache as unusable, or the load error identifies the file that failed to
parse.

Actual: the cache reports healthy at every step and the load fails with an error
that points at the accelerator rather than at the bytes.

Suggested fixes, roughly in order of value

1. Verify after download. Compare each file against the size and hash the service
   already publishes in the model manifest, and fail the download rather than the load.
   This alone would have turned a multi-day investigation into an error message.
2. Make DownloadAsync repairable. A force: true or verify: true option so an
   application can recover without reverse-engineering the cache layout.
3. Add an integrity API. Something like VerifyCacheAsync returning the offending
   files, so applications do not each have to reimplement hashing and ship their own
   reference manifests.
4. Name the file in the load failure. Wrapping ORT's message with the context
   binary path and the cache directory would let a developer reach the answer directly
   from the exception.
5. Harden the resume path. Do not let a preallocated full length file stand in for a
   complete one. Any of the following would have prevented this outright: keep the
   partial payload under a temporary name and rename only on verified completion, mark
   the destination sparse so a short valid data length is visible, record a content hash
   in .download.state and check it before declaring completion, or persist an ETag and
   revalidate on resume rather than trusting LastModified and length. Committing the
   state file and the payload writes together would also close the gap between the
   bitmap and the bytes.
6. Treat a wake from sleep as a transfer restart. The failure we hit was silent
   rather than an error, so retry logic never engaged.

Item 1 is the one that matters. The rest are mitigations for the case where it fails
anyway.

Related

The same investigation ran into a second, separate problem worth mentioning because it
compounds this one: FoundryLocalManager.IsInitialized flips to true after
CreateAsync but before DownloadAndRegisterEpsAsync completes. Touching the
catalog in that window caches CPU-only variants for the lifetime of the process, so an
NPU model silently never gets offered. We work around it by awaiting EP registration
explicitly before any catalog access. Happy to file that separately if useful.