dotnet.js cache busting (fingerprinting), Symptom: Cannot read properties of undefined (reading 'out')

[hakenr]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When upgrading from 8.0.4 to 8.0.5, the dotnet.js changed.
In opposite to dotnet.native.{fingerprint}.js and dotnet.runtime.{fingerprint}.js, the dotnet.js is not fingerprinted and thus cached. The usual Clear site data in F12-DevTools do not help as the file is cached by the browser and still taken from "(disk cache)".

The 8.0.4 dotnet.js is not compatible with 8.0.5 dotnet.native.{...}.js or dotnet.runtime.{...}.js and the symptom is:

logging.ts:32  MONO_WASM: TypeError: Cannot read properties of undefined (reading 'out')
    at $l (https://localhost:44303/_framework/dotnet.runtime.8.0.5.gongq8hbow.js:3:198913)
    at ze (https://localhost:44303/_framework/dotnet.js:3:30895)
    at https://localhost:44303/_framework/dotnet.js:3:30131
    at async Object.create (https://localhost:44303/_framework/dotnet.js:3:34510)
    at async https://localhost:44303/_framework/blazor.web.js:1:154483
    at async https://localhost:44303/_framework/blazor.web.js:1:164313

startup.ts:43  Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'out')
    at $l (startup.ts:43:17)
    at ze (run.ts:461:11)
    at run.ts:505:11
    at async Object.create (run.ts:361:72)
    at async blazor.web.js:1:154483
    at async blazor.web.js:1:164313

(No module parameter set when calling configureRuntimeStartup() from initializeModules().)

Expected Behavior

dotnet.js to be correctly fingerprinted (As it already used to be before splitting to several files?).

Steps To Reproduce

VS 2022 17.9.6 - Create new BWA - WebAssembly project + run locally in browser.
Update VS to 17.9.7 (brings dotnet 8.0.5).
Run the same project again.

Exceptions (if any)

No response

.NET Version

8.0.5 (9.0.0-preview3 still does not resolve this issue)

Anything else?

WORKAROUND: Ctrl+F5 in browser to force reloading the file.

(It seems this is already in progress, as there are some options in tests which enable checking the fingerprinting on dotnet.js.)

[hakenr]

Related:

• Blazor AOT Cache Busting sdk#20630
• 4a2f9d3

Failing lines of code:

• runtime/src/mono/browser/runtime/startup.ts

  Line 39 in 19467dc

  if (!module.out) {

• runtime/src/mono/browser/runtime/loader/run.ts

  Line 445 in 1f08a36

  await configureRuntimeStartup(emscriptenModule);

[lewing]

cc @pavelsavara

[pavelsavara]

perhaps we could somehow attach build timestamp in the blazor code
https://github.com/dotnet/aspnetcore/blob/c91ce3d2601394242866e7beae2b2339ced3a5d9/src/Components/Web.JS/src/Platform/Mono/MonoPlatform.ts#L119-L131

something like import('_framework/dotnet.js?version=' + buildId)

The other option is to bundle dotnet.js inside blazor's JS file. Assuming that it doesn't have the same issue.

cc @maraf @javiercn

[maraf]

There is a guidance about hosting and upgrading blazor apps dotnet/aspnetcore#52022 (the title mentions "major" upgrade, but in reality the situation can happen even with servicing release).

The situation will be better for hosted mode in .NET 9 thanks to dotnet/sdk#39405 and dotnet/sdk#39636.

For standalone mode ideally we should be fingerprinting everything (blazor.js, dotnet.js, all dlls/wasms, but also scripts and styles from nuget packages). But that requires rewriting code that references those files. Unlike for webpack and similar we don't have tooling to do it safely. We should review again if we can it do for the most typical scenarios

The other option is to bundle dotnet.js inside blazor's JS file. Assuming that it doesn't have the same issue.

This is unfortunately just about moving the problem to a different file

[TomGathercole]

Is there any way we can manually introduce any extra cache-busting? Ctrl+F5 is an obvious solution, but many of our customers are non-technical and they'll end up having to raise a support ticket with us to find that out.

For unrelated reasons we have to adopt patch releases of dotnet very quickly, and I'd rather avoid causing our customers grief a bunch more times before .NET 9 releases.

[maraf]

Is there any way we can manually introduce any extra cache-busting?

dotnet/aspnetcore#55751 (comment)

"Custom middleware that sets Cache-Control: no-store" for dotnet.js URL

[Bellarmine-Head]

"Custom middleware that sets Cache-Control: no-store" for dotnet.js URL

Which isn't strictly cache-busting of course... :-)

It's cache-punting.

[kg]

Is there any way we can manually introduce any extra cache-busting? Ctrl+F5 is an obvious solution, but many of our customers are non-technical and they'll end up having to raise a support ticket with us to find that out.

For unrelated reasons we have to adopt patch releases of dotnet very quickly, and I'd rather avoid causing our customers grief a bunch more times before .NET 9 releases.

My solution for this Chrome 'feature' in the past was for each deployment to go into a different versioned folder, i.e. cdn.invalid/myapp/207/ and then load the whole app from in that directory. That way paths like dotnet.js can stay hardcoded in my HTML and static files, but they don't get incorrectly cached.

Of course this isn't exactly trivial to do, but it eliminates the need to find a way to append ?buster to every URL, or manually rewrite paths in static files, or set cache-control headers. So in the interim I would suggest trying that if you can.

[maraf]

perhaps we could somehow attach build timestamp in the blazor code https://github.com/dotnet/aspnetcore/blob/c91ce3d2601394242866e7beae2b2339ced3a5d9/src/Components/Web.JS/src/Platform/Mono/MonoPlatform.ts#L119-L131

something like import('_framework/dotnet.js?version=' + buildId)

We can use loadBootResource to modify a URL resolution for dotnet.js

<script src="_framework/blazor.web.js?v=8.0.5" autostart="false"></script>
<script>
    Blazor.start({
        webAssembly: {
            loadBootResource: function (type, name, defaultUri, integrity) {
                switch (type) {
                    case 'dotnetjs':
                        return `_framework/${name}?v=8.0.5`;
                }
            }
        }
    });
</script>

In the sample I'm adding also a query string for blazor.web.js, which shouldn't be required for this particular issue. Standalone app can be modified similarly. More details in https://learn.microsoft.com/en-us/aspnet/core/blazor/fundamentals/startup?view=aspnetcore-8.0#load-client-side-boot-resources

[javiercn]

@maraf is there a chance that blazor/dotnet.js get out of "sync"? if not, we could on the Blazor side, stamp the version for the import as part of the build process, and we could then also stamp the 8.0.x on the relevant dotnet.js file as part of the build to make sure everything lines up.

[maraf]

@maraf is there a chance that blazor/dotnet.js get out of "sync"? if not, we could on the Blazor side, stamp the version for the import as part of the build process, and we could then also stamp the 8.0.x on the relevant dotnet.js file as part of the build to make sure everything lines up.

Ideally no, but blazor.js comes from a package with expicit version in csproj. Dotnet.js version might be affected by presence of workload and currentness of workload manifest. In SDK previews targeting downlevel is also not very stable.

The only 100% safe solution from my POV is to put a placeholder in blazor.js and replace it with actual version (fingerprint) of dotnet.js during app build.

[pavelsavara]

We need to make sure that the whole chain starting with HTML has ability to bust the cache. It could be hash in the file name or it could be query string.

Trying to solve this problem in the middle of the dependency chain (dotnet.js) would not work.
Because possibly the blazor.js is stale too. We need something which will bust blazor.js too, so that it could bust dotnet.js

Adding Etag to HTML file could make the validation cheap.

[maraf]

Yes, using etag + if-none-match for non-per-deployment-unique URLs and cache for ever for unique URLs is the way to go

[Bellarmine-Head]

For me the two weak points of Blazor WebAssembly have always been cache-busting and the need to set the Server project as the sole startup project in Visual Studio after creating new solution, or cloning solution from repo (since startup project choice is stored in .suo and not in .sln where it belongs).

From my client-side index.html file server-side App.razor component, the following are absolutely essential:-

<link href="css/app.css?v=14" rel="stylesheet" />   <!-- increment ?v=N when file changes -->

<link href="OutThink.ContentEditorWebsite.Server.styles.css?v=26" rel="stylesheet" />   <!-- increment ?v=N when file changes -->

<script src="js/app/app.js?v=35"></script>   <!-- increment ?v=N when file changes -->

[Bellarmine-Head]

<link href="OutThink.ContentEditorWebsite.Server.styles.css?v=26" rel="stylesheet" />

This synthesized CSS file is sent with an etag in the response headers, and I thought this was for cache-busting purposes, but imo it never worked, and I've always had to set the ?v=N query string.

[maraf]

This synthesized CSS file is sent with an etag in the response headers, and I thought this was for cache-busting purposes, but imo it never worked, and I've always had to set the ?v=N query string.

AFAIK the problem is that it does set etag, but it doesn't set cache-control and thus it's up to the browser. Whereas for blazor.js it does add cache-control: no-store and thus a revalidate request is issued by the browser

app.css

blazor.web.js

[Bellarmine-Head]

AFAIK the problem is that it does set etag, but it doesn't set cache-control and thus it's up to the browser.

Ok, that would explain it.

Re. index.html / App.Razor, I've never had a problem with the browser caching the Blazor WebAssembly-served HTML. E.g. all changes I make here are immediately picked up, and I've never had to worry about having to hit ctrl+F5 or anything like that.

[maraf]

From my testing it has always served HTML with cache-control: no-cache, no-store

[javiercn]

@double-dubs that file is not modified by the build, it's just copied, so it makes sense for it to have an older timestamp. That said, hashes are the way to know for sure what's going on.

[double-dubs]

Ok, the hash did NOT match on the dotnet.js file. I did a hash on the blazor.webassebly.js file and that one matched. Is Azure static web apps doing some caching on that file? I looked in the portal UI and don't see any options to clear the cache.

[pingu2k4]

Hey,

The files when locally published to folder, versus published to SWA through our azure pipeline differs.

I made a diff in case that is useful...
(Local publish on the left, SWA published file on the right)

[maraf]

Based on the git hashes in the provided diff, your local dotnet.js is 8.0.0 and Azure SWA is 8.0.3, which is very odd based on the screenshot in comment above showing dotnet.runtime.8.0.5.*.js

[pingu2k4]

Running dotnet --version locally I get 8.0.205, and not sure what would be running on our pipelines as we don't directly specify, but one of the differences that I noted between the last successful deployment and deploying the same commit once it started failing, so it bringing in 8.0.5 related stuff.

A small example, but on the left in this screenshot was the azure devops pipeline logs from the last successful build, and on the right is deploying the same commit once it started failing.

[maraf]

The 8.0.205 contains 8.0.4 runtime, so it's even more odd that your local publish dir contains 8.0.0. Can you please publish again -bl and share a msbuild.binlog (either here or through a ticket at https://developercommunity.visualstudio.com/ with MSFT only visible attachment)

[maraf]

Brotli compressed dotnet.js asset can get stale with incremental publish. It happens when the publish folder already contained previously published assets (so not typical CI scenario, but it could have affect on manual publish). Brotli compressed assets are computed only for publish, during build they are not computed at all.

Setup

• Have SDK 8.0.204
• Have SDK 8.0.300

Repro

1. Create new project (web with wasm interativity or wasm standalone)
2. Publish with older SDK (8.0.204)
3. Publish with newer SDK (8.0.300)

Output in bin\Release\net8.0\publish\

• dotnet.js is updated
• dotnet.js.gz is updated
• dotnet.js.br is stale

msbuild.binlog.zip (the asset is obj\Release\net8.0\\compressed\publish\maj14cmhj5.br is not recomputed with second publish)

Skipping 'D:\Development\samples\BuildIncrementalismSdkUpgrade\BlazorWeb\BlazorWeb.Client\bin\Release\net8.0\wwwroot\_framework\dotnet.js' because 'obj\Release\net8.0\\compressed\publish\maj14cmhj5.br' is newer than 'D:\Development\samples\BuildIncrementalismSdkUpgrade\BlazorWeb\BlazorWeb.Client\bin\Release\net8.0\wwwroot\_framework\dotnet.js'.

For build compression, the file hash is computed from runtime pack location, and so it changes when runtime pack version changes (because of different location). For publish compression, the file hash is computed from build output path (eg. bin\Release\net8.0\wwwroot\_framework\dotnet.js), and so it doesn't change when runtime pack version changes. With BuildCompressionFormats="" the same issue manifests also for GZip.

This logic seems to be correct only when "runtime pack file" is newer than "last build compressed file" (the same for GZip) https://github.com/dotnet/sdk/blob/main/src/StaticWebAssetsSdk/Tasks/Compression/BrotliCompress.cs#L93

It can probably happen for any asset coming directly from runtime pack and not modified during publish. Assemblies are now transformed to webcil, ICU and all JS files might be probably affected.

[LaughingJohn]

I've just fired up my Blazor App in VS2022 preview, I think for the first time since upgrading to 17.11 preview 1.0, and have got this which I think might be related? (question mark because I'm not sure I understand the issue). This is just running it up in debug. I tried deleting the bin & obj folders, but that made no difference.

MONO_WASM: TypeError: Cannot read properties of undefined (reading 'out')
    at $l (https://localhost:44376/_framework/dotnet.runtime.8.0.5.vjjqs9rnu5.js:3:198913)
    at ze (https://localhost:44376/_framework/dotnet.js:3:30895)
    at https://localhost:44376/_framework/dotnet.js:3:30131
    at async Object.create (https://localhost:44376/_framework/dotnet.js:3:34510)
    at async https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:43466
    at async https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:58010
    at async mn (https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:57613)
Error in mono_download_assets: TypeError: Cannot read properties of undefined (reading 'out')
TypeError: Cannot read properties of undefined (reading 'out')
Stack trace:
 >  at $l (https://localhost:44376/_framework/dotnet.runtime.8.0.5.vjjqs9rnu5.js:3:198913)
 >    at ze (https://localhost:44376/_framework/dotnet.js:3:30895)
 >    at https://localhost:44376/_framework/dotnet.js:3:30131
 >    at async Object.create (https://localhost:44376/_framework/dotnet.js:3:34510)
 >    at async https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:43466
 >    at async https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:58010
 >    at async mn (https://localhost:44376/_framework/blazor.webassembly.js?v=gvHfnndfEu1tRf0rFb5988rWq7ITIotOaE8-AMbKYbc:1:57613)

[steamonimo]

Same problem but with a twist: It runs in development but deployed to the same machine it does not run. It makes no difference if the cache is cleared or not:

http://localhost/_framework/dotnet.runtime.8.0.5.8hv7wyhg5a.js:3:198913
NO_WASM: TypeError: Cannot read properties of undefined (reading 'out')

I find it interesting that after the installation of the visual studio update (17.10.1) AND hosting-package for 8.0.6 the command dotnet --version does not give me 8.0.6 but 8.0.300.

What a bummer. My blazor based release pipeline came to a total standstill and client is not amused. Uninstalling visual studio update 17.10.1 now. Same problem with 17.10.0 and - another bummer - you can not go back to 17.9.x !

[maraf]

@steamonimo What do you mean by deployed? Are you doing it from VS or command line? Can you produce a binlog and share it (how to here https://msbuildlog.com)?

[steamonimo]

@maraf Here are the binlog files. Please let me know if you need more!

[maraf]

@steamonimo Sorry for late reply. From the build log is visible that it skipped copying dotnet.js as the build has decided it's new enough. Do you have by any change still the build output on disk? Can you upload the _framework/dotnet.js as well?

[steamonimo]

@maraf In the binlog files under wwwroot/_framework you will find all files in the build output starting with dotnet.js.

[maraf]

@steamonimo Thank you! The problem seems to be the one described in #102272 (comment) - brotli compressed file is stale. We are tracking that one. It can be workarounded by cleaning the publish dir before doing the next one, as publish is not incremental at the moment (as you can see on your disk)

[steamonimo]

@maraf I used a new publication folder. After rebuild there are less dotnet.js related files (see binlog files but the problem remains the same. For git the remaining files are the same files as the previous commit (unchanged).

Detail: the error in the browser log is now about the digest which is new (see readme):

Failed to find a valid digest in the 'integrity' attribute for resource 'http://127.0.0.1/_framework/dotnet.native.wasm' with computed SHA-256 integrity 'U1JSgnum2OTerSq8i5Y9L17deHEWJmae0djinSPLyJM='. The resource has been blocked.

However, the status code for dotnet.native.wasm is 200 (OK). The problem must be digest related...

Added: blazor.boot.json for digest analysis in binlog/wwwroot/_framework

[steamonimo]

@maraf

However, the status code for dotnet.native.wasm is 200 (OK). The problem must be digest related...

The tool integrity.ps1 does also indicate that dotnet is the root of the problem. All files mentioned below and the blazor.root.json with the digest info can be found in my binlog.

Invoke-WebRequest: integrity.ps1:112
Line |
112 | … viceWorkerAssetsResponse = Invoke-WebRequest $serviceWorkerAssetsUrl;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Response status code does not indicate success: 404 (Not Found).
Signatures for 'wwwroot/_framework/dotnet.js (zLPXeNQlcFjOiHTxq90o/aQv1TJK3xZIBQpEXrMM7QM=)' and 'wwwroot/_framework/dotnet.js.br (XRJbbIo8kGIKeCz+wbs1aiCrjY5+TWBOyy4soMDhBRE=)' do not match.
Signatures for 'wwwroot/_framework/dotnet.native.wasm (b3QtgAqvT5FoNyYgoEYsdE4zF5C1Is6lhQ8VEkROor8=)' and 'wwwroot/_framework/dotnet.native.wasm.br (U1JSgnum2OTerSq8i5Y9L17deHEWJmae0djinSPLyJM=)' do not match.

[steamonimo]

The problem persists with Visual Studio 17.10.2. The binlog has been updated...

[steamonimo]

@maraf The problem also persists with Visual Studio 17.10.3. The binlog has been updated. The error in the browser changed to:

Failed to find a valid digest in the 'integrity' attribute for resource 'http://127.0.0.1/_framework/dotnet.native.wasm' with computed SHA-256 integrity 'U1JSgnum2OTerSq8i5Y9L17deHEWJmae0djinSPLyJM='. The resource has been blocked.

[stagep]

@steamonimo
Same issue here but it was resolved after seeing dotnet/aspnetcore#55751 (comment)
that says to delete obj and bin folders before deploying and clearing cache on browser.

[steamonimo]

delete obj and bin folders before deploying and clearing cache on browser.

@stagep That is the solution! Many thanks for your help!
@maraf I would be nice to see these things fixed in the VS tool chain.

[zubairkhakwani]

Done everything but was unable to make it work, deleted the bin and obj twice before rebuilding and publishing & cleared the
browser cache as well. but the same error persists.

Please if there is any workaround mention.

I am using blazor WebApp and my app keep running on server & unable to start WASM due to this website hangs because to slow internet.

[LeeStevens318]

Any update? I'm using Git Actions to Build and Deploy no bin/obj Folders to delete.

[maraf]

@LeeStevens318 Could you please collect a binlog from your CI build/publish and share it either here or through https://developercommunity.visualstudio.com/? Does your runner cache in anyway build outputs between runs?

The only issue we have been able to find so far is related to incremental publish.

[nicholasfortier]

Had the same issue, no bin/obj to delete. Our CI was targeting the latest SDK. Fixed it by targeting 8.0.204 (for now).

[maraf]

@nicholasfortier Could you please collect a binlog from your CI build/publish and share it either here or through https://developercommunity.visualstudio.com/? Does your runner cache in anyway build outputs between runs?

[andreimilto]

The issue persists when publishing to a folder from Visual Studio v17.11.2 (dotnet --version 8.0.400): files dotnet.js and dotnet.js.gz get updated, but dotnet.js.br gets stuck on the old version.
Cleaning up bin and obj directories before publishing fixes the problem.

[omni-htg]

Clearing the bin and obj did not help in my case, seeing this issue in 8.0.10 / 8.0.403 now.
It shows up as a different message but it might be related to this one, see dotnet/aspnetcore#58313 above.

[maraf]

@omni-htg Did you updated SDK after last publish? Can you please collect binlog from publish and share it here or though https://developercommunity.visualstudio.com ?