.net 9 Blazor Server application pool recycle issue

[wstaelens]

Since we've upgraded to .net 9 (coming from .net 8) we experience that some of our customers can't click on any buttons/links/.. being rendered in Blazor Server. The click events are not being accepted/processed.

Server 2019, Blazor Server, .net 9.0 in-process published, IIS.

We've seen that when we see an Event id 5074 WAS in that the application pool recycles that some users in production are affected by the "ignoring of click events". These are not always the same users.
These users are not being able to work in the applicaton as the click events are totally ignored.

We've never seen this in earlier versions of .net but since .net 9 we experience this sometimes in production.

Anyone experienced the same? Suggestions?

When a customer calls with the issue, the CPU (7%, 6%) and Memory (70%) are normal...

[danroth27]

@wstaelens When the app pool recycles, any active Blazor Server circuits will be lost, so while the app is recycling there is nothing to handle the UI events. Blazor should detect that the connection to the server was lost, and once the server comes back up, Blazor should try to reconnect and then refresh the browser to establish a new circuit. Are the users not seeing this disconnected state?

[wstaelens]

@wstaelens When the app pool recycles, any active Blazor Server circuits will be lost, so while the app is recycling there is nothing to handle the UI events. Blazor should detect that the connection to the server was lost, and once the server comes back up, Blazor should try to reconnect and then refresh the browser to establish a new circuit. Are the users not seeing this disconnected state?

@danroth27 the users don't see the disconnected state.
If for example 40 users are active, 35 or 38 users have no issue but just a few seem to sometimes have an issue.
It are also not always the same user(s) in case it happens.

We never had this issue in versions <= .net 8. Only started happening in .net 9 in production.

Yesterday it happened "luckily" with a colleague his account.
He had one browser open on a page rendered with blazor server, but could not click on anything.
Nothing in browsers developers console related to javascript or something...

He copied the url and tried different browsers, normal and incognito, tried F5, ctrl+F5, etc... but no success or no reconnection.

Is there anything we can check/debug or do?

Our solution has grown over the years we started with the first versions of Blazor Server / .net core and evoluated till today Blazor Server and .net 9.
Maybe something "old" is still in the solution, conflicting, but we don't know what that could be.

[wstaelens]

@danroth27 some extra:

Foo.razor

has a button (based on DevExpress DXButton) with a Click event:

Click="(async x => await OpenFoo(fooInfo.Foo))"

This calls:

protected async Task OpenFoo(FooDto f) {
   await csession.OpenFoo(new OpenFooCommand { someID = f.Id });
}

which calls:

    public async Task<FooResultDto> OpenFoo(IOpenFooCommand cmd, bool autoNavigate = true, bool handleErr = true)
    {
      var loadingDlg = LoadingDialog.ShowDialog(_ModalSrvc, null, "Please wait", "logo.svg");
      await Task.Yield(); 
      // ...
    }

We can't verify but we think that this is the case or similar

1. website is open where Foo.razor is shown
2. recycle happens
3. User is able to click on a button which calls the Click --> OpenFoo(FooDto --> OpenFoo(IOpenFooCommand
4. We think that await Task.Yield() is blocking everything for that user/session.
   but for the user itself it feels like the click never happened because the UI doesn't respond (clicks are not registered). If they click on this they cannot click anything else.

This code never had issues in .net 8 but obviously seems to block in .net 9 somewhere related to recycles.

in .net 6 or 7 we had an issue that a dialog was not appearing/rendering and we added in the past await Task.Yield() so that the loading dialog would aways be visible. That is why the Task.Yield() has been added.
(based on this documentation: https://learn.microsoft.com/en-us/aspnet/core/blazor/components/rendering?view=aspnetcore-8.0#statehaschanged )

does that make any sense to you?

[javiercn]

@wstaelens thanks for the additional details.

It makes sense in a very limited way, given that we don't have the context. The first thing we would suggest is turning logging to debug level both on the server and on the client to try and determine if there are exceptions or other types of errors happening.

It's unclear what var loadingDlg = LoadingDialog.ShowDialog(_ModalSrvc, null, "Please wait", "logo.svg"); is doing, so we don't know if that's causing an error that tears down the entire process (and forces the pool to recycle) or if that's something that is literally blocking the thread for that user.

Task.Yield() is unlikely to block anything, as it literally yields the thread of execution to give other things time to execute.

We would suggest that you introduce some state change on the UI before await csession.OpenFoo(new OpenFooCommand { someID = f.Id }); (for example, disable the button clicked) and use Task.Yield() before that call. So for example:

// Some status update
await Task.Yield();
var task = csession.OpenFoo(new OpenFooCommand { someID = f.Id });
// Some other status update
await Task.Yield();
// Some other status update
await task;
// Some other status update

[wstaelens]

we'll give it a try with the await Task.Yield(); before the dialog.

The dialog is from Blazored.Modal: https://github.com/Blazored/Modal