A benchmark job with a Native AOT tool chain fails on package restore when package source mapping is used

[akamsteeg]

(BenchmarkDotNet v0.14 on Windows 11.)

I have a solution that uses NuGet package source mapping. There's a benchmark project with a Program.cs that looks like this:

public static class Program
{
  public static void Main(string[] args)
  {
    var config = GetConfig();
    BenchmarkSwitcher
      .FromAssembly(Assembly.GetExecutingAssembly())
      .Run(args, config);
  }

  private static ManualConfig GetConfig()
  {
    var job = Job.Default;

    var config = ManualConfig.Create(DefaultConfig.Instance)
      .AddJob(
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
        Job.Default.WithRuntime(NativeAotRuntime.Net80),
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp60)
        )
      .AddDiagnoser(MemoryDiagnoser.Default);

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithToolchain(CsProjClassicNetToolchain.Net481));
    }

    config.SummaryStyle = SummaryStyle.Default
      .WithRatioStyle(RatioStyle.Percentage);

    config.AddValidator(JitOptimizationsValidator.FailOnError); // Fail when any of the referenced assemblies are not optimized

    config.WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest));

    return config;
  }
}

So I'm running benchmarks against .NET 8 as my baseline, .NET 8 with Native AOT, .NET 6 and .NET 4.8.1 when on Windows. When running this using dotnet build -c Release && dotnet run -c Release -f net8.0 the console shows this for the "Native AOT on .NET 8" job:

// Build Error: Standard output:

 Standard error:
   Determining projects to restore...
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.DotNet.ILCompiler (>= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.DotNet.ILCompiler (>= 8.0.5)' for 'net8.0/win-x64'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'runtime.win-x64.Microsoft.DotNet.ILCompiler (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
  Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj (in 188 ms).
  Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj (in 188 ms).
  Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj (in 205 ms).

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

This also happens when there's only one job with the NativeAotRuntime.Net80 tool chain and not the others.

I can solve this by changing my nuget.config and add a package source mapping with that nativeAotNuGetFeed name:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="NuGet.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>

  <packageSourceMapping>
    <packageSource key="NuGet.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="nativeAotNuGetFeed">  <!-- Hack for getting Native AOT benchmarks working in the AtleX.HaveIBeenPwned.Benchmarks project -->
      <package pattern="*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

This allows the benchmarks to run successfully. And after it has run once on a machine, I can remove the packageSource mapping for nativeAotNuGetFeed again and it will continue to run. However, it will fail again on any machine that hasn't run the Native AOT benchmarks before.

For reproduction the code I ran is on Github: akamsteeg/AtleX.HaveIBeenPwned@c6c6cd4. Running it with dotnet build -c Release && dotnet run -c Release -f net8.0 and picking any benchmark should reproduce this. ⚠️ Remember that the benchmark continues to succeed on the same machine once it has managed to restore the packages once. Probably clearing package caches will help, but I haven't tried that.

[akamsteeg]

I just noticed this part of the log:

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6

I managed to secure the contents of that C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6 folder and zip it up.

You can find it here:
bin.zip