TypeLoadExceptions on .NET 9 when referencing HttpSys on Linux

[MihaZupan]

YARP will try to grab the IServerDelegationFeature on startup to detect if the functionality is supported.
_serverDelegationFeature = server.Features?.Get<IServerDelegationFeature>();

We're seeing many users run into TypeLoadExceptions when upgrading their app from .NET 8 to 9:
dotnet/yarp#2652

System.TypeLoadException: Could not load type 'Microsoft.AspNetCore.Server.HttpSys.IServerDelegationFeature' from assembly 'Microsoft.AspNetCore.Server.HttpSys, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
   at Yarp.ReverseProxy.Delegation.HttpSysDelegator..ctor(IServer server, ILogger`1 logger)

Reports mention both Ubuntu 22 and 24.
I was not able to reproduce the issue myself (on a VM nor WSL).

Possibly a recent regression since RC2?
dotnet/yarp#2652 (comment)

To give some extra confusion, we've been running our test envoirment on 9RC1 and 9RC2 when they came out, and didn't have the issue, they only came with the official release (I use ./dotnet-install.sh, no apt since it wasn't released there yet).

---

What sort of information would be needed to help diagnose this?

[Tratcher]

@HugoVG, @dario-l, @ebesavara, @mikoskinen, @shainegordon, can any of you provide a repro project and steps? Preferably as a github repo as this kind of issue depends a lot on the project file, dependencies, publish settings, etc.

[HugoVG]

making one right now

[HugoVG]

https://github.com/HugoVG/Dotnet9-linux-crash-sanityCheck

I put it in here with my stack trace
(keep in mind I've only grabbed the wsl one, I didn't have the other server at the moment)
WSL: Ubuntu 22.04
Test Servers: Ubuntu 22.04

Downloaded with ./dotnet-install.sh

[Tratcher]

Possible culprit: http://archive.ubuntu.com/ubuntu/pool/universe/d/dotnet9/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb
https://ubuntu.pkgs.org/24.10/ubuntu-updates-universe-amd64/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb.html

This deb package has an empty Microsoft.AspNetCore.Server.HttpSys.dll and different (invalid?) commit shaw a2bc464e40415d625118f38fbb0556d1803783ff. All the other versions I've seen have commit shaw af22eff.

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

	<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
	<Compile Remove="**\*.cs" />
	</ItemGroup>

[sebastienros]

@richlander

[shainegordon]

Possible culprit: http://archive.ubuntu.com/ubuntu/pool/universe/d/dotnet9/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb https://ubuntu.pkgs.org/24.10/ubuntu-updates-universe-amd64/aspnetcore-runtime-9.0_9.0.0-rtm-0ubuntu1~24.10.1_amd64.deb.html

This deb package has an empty Microsoft.AspNetCore.Server.HttpSys.dll and different (invalid?) commit shaw a2bc464e40415d625118f38fbb0556d1803783ff. All the other versions I've seen have commit shaw af22eff.

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

I’m on 24.04, so am using the backports PPA to install .NET 9, rather than the one in the 24.10 distribution - but I’m not clued up as to how backports actually work

[adityamandaleeka]

Tagging @mthalman @MichaelSimons since this is likely source-build related based on the comments above.

[mthalman]

Was this deb package built from source?

aspnetcore/src/Servers/HttpSys/src/Microsoft.AspNetCore.Server.HttpSys.csproj

Lines 36 to 38 in 764eb46

<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'"> 
  <Compile Remove="**\*.cs" /> 
</ItemGroup>

Yes, the deb package would have been built from source with DotNetBuildSourceOnly set to true.

Looking at the assembly, I can confirm that it is empty with no classes defined. So that matches with how the project is configured here. This was removed in #50685. cc @Tratcher

[mthalman]

Since this is a result of the configuration of the project file, folks from the ASP.NET Core team should investigate further. The source build team doesn't have the knowledge of how this should be resolved. But we're happy to help with any questions.

[Tratcher]

So why did it work for people in 9.0-rc2 and not rtm? Were they not consuming the source build package?

[mthalman]

Ubuntu never provided DEB packages until 9.0 GA. So any repro using RC2 was using a Microsoft-built installation (non-source build).

[adityamandaleeka]

Thanks @mthalman. I looked at the history here and it looks like we excluded this code from source build at the start of 9.0 because (a) it was being changed to include an external dependency on CsWin32 which isn't allowed in source build and (b) the logical assumption was that HttpSys wouldn't be relevant on non-Windows platforms anyway.

Given that the latter is evidently not true, we need to decide what to do about it while avoiding breaking changes.

It is unfortunate that this scenario never got any mileage on previews/RCs since those were using the MS-built bits...

@wtgodbe @Tratcher What do you think?

[wtgodbe]

it was being changed to include an external dependency on CsWin32 which isn't allowed in source build and (b) the logical assumption was that HttpSys wouldn't be relevant on non-Windows platforms anyway.

Not knowing a lot about this myself, shouldn't CsWin32 not be relevant on non-windows? That is, can we just condition the usage of CsWin32 on the build being on windows, rather than the build being DotNetSourceBuildOnly?

[BrennanConroy]

The usage of CsWin32 is deeply ingrained in the build of Http.Sys. You can't just not use it. What it seems like we need is empty (throwing?) implementations of all public API in Http.Sys. Like https://github.com/dotnet/aspnetcore/pull/50685/files#diff-bd7fe97e25b6f14c0848751571af0c6f14be592288322b7fb8f5ddb44ce1a3d8R17
That does seem like a pain to maintain though, if we add new API we'd need to remember to make a noop version just for source-build...

[richlander]

Another key aspect is how we get earlier test coverage. It is possible that Canonical will offer .NET 10 previews via their PPA and/or snap. That would help a lot. I suspect that many people (like me) would happily choose that option over tarballs.

[Tratcher]

@richlander, the first thing we need is a mitigation for 9.0.0. Can you give instructions for people to install a non-sourcebuild version of dotnet on ubuntu?

[Tratcher]

The usage of CsWin32 is deeply ingrained in the build of Http.Sys. You can't just not use it. What it seems like we need is empty (throwing?) implementations of all public API in Http.Sys. Like https://github.com/dotnet/aspnetcore/pull/50685/files#diff-bd7fe97e25b6f14c0848751571af0c6f14be592288322b7fb8f5ddb44ce1a3d8R17 That does seem like a pain to maintain though, if we add new API we'd need to remember to make a noop version just for source-build...

Yes, though the public API surface area is pretty minimal. It will be the same as the ref assemblies.

[richlander]

Users can following the "manual install" instructions at: https://learn.microsoft.com/dotnet/core/install/linux-ubuntu

Same thing as downloading a tarball from https://dotnet.microsoft.com/download/dotnet/9.0

We don't offer packages for Ubuntu, per dotnet/core#9556.

[HugoVG]

Users can following the "manual install" instructions at: https://learn.microsoft.com/dotnet/core/install/linux-ubuntu

Same thing as downloading a tarball from https://dotnet.microsoft.com/download/dotnet/9.0

We don't offer packages for Ubuntu, per dotnet/core#9556.

Which is weird, I migrated my application to dotnet 9 before there was an Ubuntu package, following the install with the ./dotnet-install.sh (time stamp of my installation file Nov 13 07:50 dotnet-install.sh) and I'm running into the same issues the people who have the Deb package have.

[richlander]

Thanks for sharing. That suggests a broader problem.

As you no doubt realize, the install script installs the Microsoft build of .NET.

[shainegordon]

Thanks for sharing. That suggests a broader problem.

As you no doubt realize, the install script installs the Microsoft build of .NET.

I dont think this is an issue with the install script versions.

Last week I migrated away from Ubuntu managed .NET (I wanted the lastest .NET 8 SDK), and as such have installed .NET 9 via the dotnet-install.sh script, and I do not have this Yarp issue as I originally did when using APT + Backport

➜ dotnet --list-sdks             

8.0.404 [/home/shaine/.dotnet/sdk]
9.0.100 [/home/shaine/.dotnet/sdk]

[MihaZupan]

I can only reproduce this with the dotnet-sdk-9.0 package, not on any build from dotnet-install.sh, so the source build suspicion appears to be correct.

[mthalman]

Another key aspect is how we get earlier test coverage. It is possible that Canonical will offer .NET 10 previews via their PPA and/or snap. That would help a lot. I suspect that many people (like me) would happily choose that option over tarballs.

Please consider whether it would be appropriate to add a test which verifies this scenario to https://github.com/dotnet/scenario-tests. Those tests get executed with the source built SDK for each build.

[Tratcher]

@mthalman how about at test that ensures all the public API are the same between source build outputs and full builds? Or between the source build and the ref assemblies? That would catch this whole class of problems.

[mthalman]

@mthalman how about at test that ensures all the public API are the same between source build outputs and full builds? Or between the source build and the ref assemblies? That would catch this whole class of problems.

Logged dotnet/source-build#4795

[ebesavara]

We have a new version of .NET 9, a patch release that was recently released, and yet the issue has not been resolved. This is blocking us from deploying .NET 9 to production with our projects.

[joelmandell]

Yes experiencing same issue on Fedora 41 and 9.0.101 [/usr/lib64/dotnet/sdk]
(Unhandled exception. System.TypeLoadException: Could not load type 'Microsoft.AspNetCore.Server.HttpSys.IServerDelegationFeature' from assembly 'Microsoft.AspNetCore.Server.HttpSys, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.)

It is not so nice :(. But thank you for looking into this

[MihaZupan]

FWIW there are two ways to workaround this issue, so it shouldn't be totally blocking:

• Install .NET via dotnet-install.sh instead: https://learn.microsoft.com/dotnet/core/tools/dotnet-install-script
• Use a daily build of YARP: https://github.com/microsoft/reverse-proxy/blob/main/docs/DailyBuilds.md

[joelmandell]

Thx @MihaZupan I went with the daily builds for now :=)

[omajid]

Hey @Tratcher is there a fix for this in the works?

[adityamandaleeka]

@omajid We plan to ship another YARP release with the workaround that's in the daily builds. It doesn't fix the underlying issue but will solve this problem for YARP users.

[omajid]

Thanks, @adityamandaleeka !

Am I understanding it correctly that Microsoft.AspNetCore.Server.HttpSys.dll being empty is acceptable and wont be addressed for 9.0? And anything (such as YARP) that relies on that should be fixed instead?

[adityamandaleeka]

@omajid I'd like us to get to the bottom of that as well (and the process issues that led to this not being discovered in time to resolve it before 9.0). However, those things will certainly take more time than this workaround in YARP so I think it's best to start with this.

It seems like this is probably the most likely case people will run into this issue anyway... there's not many other reasons why someone would be trying to do anything related to HTTP.sys if they're on Linux.

[Tratcher]

The missing APIs mean in 9.0 it's impossible to write a cross-platform app or library (like YARP) that uses Http.Sys on Windows and Kestrel on Linux, because even referencing Http.Sys causes this crash on Linux. I don't know how common that is, but some basic fix should be made in 9.0. YARP users noticed really fast...