Merge runtime async support into main (#79833)

* Handle basic await scenarios (#76121)

Add initial handling of expressions that return `Task`, `Task<T>`, `ValueTask`, `ValueTask<T>`.

* Add RuntimeAsyncMethodGenerationAttribute (#77543)

Adds control for whether to use runtime async. The flowchart is as follows:

1. The flag `System.Runtime.CompilerServices.RuntimeFeature.Async` must be present.
2. Assuming that flag is present, we look for the presence of `System.Runtime.CompilerServices.RuntimeAsyncMethodGenerationAttribute` on the method. If that attribute is present, we use the preference expressed in the attribute. The preference does not carry to nested contexts, such as local functions or lambdas.
3. If the attribute is not present, we look for `features:runtime-async=on` on the command line. If that is present, then the feature is on by default. Otherwise, the feature is off.

* Semantic search info

* Implement custom awaitable support (#78071)

This adds support for awaiting task-like types that are not natively supported by runtime async. Closes #77897.

* Move runtime async method validation into initial binding (#78310)

We now do method construction and validation for runtime async helpers up front in initial binding, rather than doing it in `RuntimeAsyncRewriter`. I've also renamed the APIs as per dotnet/runtime#114310 (comment) (though I haven't added ConfigureAwait support yet, that will be the next PR). We now validate:

* The helpers come from `System.Runtime.CompilerServices.AsyncHelpers`, defined in corelib. This means that I now need a fairly extensive corelib mock to be able to compile. When we have a testing runtime that defines these helpers, we can remove the giant mock and use the real one.
* We properly error when expected helpers aren't present.
* We properly check to make sure that constraints are satisfied when doing generic substitution in one of the runtime helpers.
* Runtime async is not turned on if the async method does not return `Task`, `Task<T>`, `ValueTask`, or `ValueTask<T>`.

Relates to test plan #75960

* React to main changes #78246 and #78231.

* Switch MethodImplAttributes.Async to 0x2000 (#78536)

It was changed in dotnet/runtime#114310 as 0x400 is a thing in framework.

* Ensure return local is the correct type for runtime async (#78603)

* Add test demonstrating current behavior

* Ensure return local is the correct type in async scenarios

* Ensure method is actually async when doing local rewrite

* Exception Handler support (#78773)

* Update EH tests to run with runtime async

* Handle non-null exception filter prologues in the spill sequencer

* Add more testing to show current incorrect behavior

* Unskip ConditionalFacts that do not need to be skipped.

* Handle ensuring that the method remains valid, even when there is an `await` in a finally section. Add signifcant testing of `await using`.

* Fix baselines

* Support `await foreach` and add runtime async verification to existing tests.

* Remove unnecessary generic

* Failing tests, add async void test suggestion

* CI failures

* Add additional test

* Test fixes

* Remove implemented PROTOTYPE, add assertion on behavior.

* Update to SpillSequenceSpiller after some more debugging and tightening the assertion

* Fix nullref

* Enable nullable for VisitCatchBlock

* Support using a simple overload resolution for finding Await helpers from the BCL (#79135)

* Support using a simple overload resolution for finding Await helpers from the BCL

This PR removes special knowledge of what `Await` helpers correspond to what types, and instead implements a very simple form of overload resolution. We immediately bail on any conflict or error and fall back to attempting to use `AwaitAwaiter` or `UnsafeAwaitAwaiter` when such scenarios are detected. I've also updated the rules to better reflect what is actually implementable.

* Create the full BoundCall in initial binding.

* PR feedback.

* Baseline struct lifting tests (#79505)

* Extract expectedOutput constants, minor touchups

* Rename expected -> expectedOutput

* Include new testing with placeholder baselines

* Progress

* First ILVerify pass

* Initial baseline IL run.

* Further baseline additions and skips based on missing corelib apis.

* Clone async void tests and have them use async Task, and validate existing code spit for these under runtime async

* Update baselines after .NET 10 intake

* Delete the stub

* Remove long dynamic baseline and leave a prototype.

* Feedback.

* BOM

* Remove unused references parameter

* Block `await dynamic`

* Block hoisted variables from runtime async for now

* Update test baselines for block

* Block arglist in runtime async

* Add IL baseline

* Handle an additional branch beyond the end of the method case.

* Move prototype comments to issues.

* Remove entry point prototypes

* Add assert and comment

* Add back assert

* Report obsolete/experimental diagnostics on await helpers.

* Fix ref safety analysis build error.

---------

Signed-off-by: Emmanuel Ferdman <[email protected]>
Co-authored-by: Jan Jones <[email protected]>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Najmabadi <[email protected]>
Co-authored-by: David Barbet <[email protected]>
Co-authored-by: Ankita Khera <[email protected]>
Co-authored-by: David Wengier <[email protected]>
Co-authored-by: Rikki Gibson <[email protected]>
Co-authored-by: Cyrus Najmabadi <[email protected]>
Co-authored-by: Rikki Gibson <[email protected]>
Co-authored-by: akhera99 <[email protected]>
Co-authored-by: Joey Robichaud <[email protected]>
Co-authored-by: Todd Grunke <[email protected]>
Co-authored-by: Youssef1313 <[email protected]>
Co-authored-by: Joey Robichaud <[email protected]>
Co-authored-by: Tomáš Matoušek <[email protected]>
Co-authored-by: Amadeusz Wieczorek <[email protected]>
Co-authored-by: Charles Stoner <[email protected]>
Co-authored-by: Jared Parsons <[email protected]>
Co-authored-by: Sam Harwell <[email protected]>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Jason Malinowski <[email protected]>
Co-authored-by: Etienne Baudoux <[email protected]>
Co-authored-by: AlekseyTs <[email protected]>
Co-authored-by: Jan Jones <[email protected]>
Co-authored-by: Maryam Ariyan <[email protected]>
Co-authored-by: Andrew Hall <[email protected]>
Co-authored-by: Arun Chander <[email protected]>
Co-authored-by: Kauwai Lucchesi <[email protected]>
Co-authored-by: Bill Wagner <[email protected]>
Co-authored-by: PaddiM8 <[email protected]>
Co-authored-by: Matteo Prosperi <[email protected]>
Co-authored-by: Julien Couvreur <[email protected]>
Co-authored-by: Matteo Prosperi <[email protected]>
Co-authored-by: Carlos Sánchez López <[email protected]>
Co-authored-by: Tomas Matousek <[email protected]>
Co-authored-by: Deepak Rathore (ALLYIS INC) <[email protected]>
Co-authored-by: Emmanuel Ferdman <[email protected]>
Co-authored-by: Evgeny Tvorun <[email protected]>
Co-authored-by: Victor Pogor <[email protected]>
Co-authored-by: Ella Hathaway <[email protected]>
Co-authored-by: Viktor Hofer <[email protected]>
Co-authored-by: Jason Malinowski <[email protected]>
Co-authored-by: DoctorKrolic <[email protected]>
Co-authored-by: John Douglas Leitch <[email protected]>
Co-authored-by: Matt Thalman <[email protected]>
Co-authored-by: Bernd Baumanns <[email protected]>
Co-authored-by: Thomas Shephard <[email protected]>
Co-authored-by: DoctorKrolic <[email protected]>
Co-authored-by: David Barbet <[email protected]>
Co-authored-by: Chris Sienkiewicz <[email protected]>
Co-authored-by: tmat <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Gen Lu <[email protected]>
Co-authored-by: Oleg Tkachenko <[email protected]>
Co-authored-by: Matt Mitchell <[email protected]>
Co-authored-by: Djuradj Kurepa <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Stuart Lang <[email protected]>
Co-authored-by: RaymondY <[email protected]>
Co-authored-by: Gobind Singh <[email protected]>
Co-authored-by: David Kean <[email protected]>

Author: 49 people

docs/compilers/CSharp/Runtime Async Design.md

@@ -57,7 +57,7 @@ namespace System.Runtime.CompilerServices;
 
 public enum MethodImplOptions
 {
-    Async = 1024
+    Async = 0x2000
 }
 ```
 
@@ -135,9 +135,11 @@ For any `await expr` with where `expr` has type `E`, the compiler will attempt t
       2. There is an identity or implicit reference conversion from `E` to the type of `P`.
    4. Otherwise, if `Mi` has a generic arity of 1 with type param `Tm`, all of the following must be true, or `Mi` is removed:
       1. The return type is `Tm`
-      2. There is an identity or implicit reference conversion from `E`'s unsubstituted definition to `P`
-      3. `E`'s type argument, `Te`, is valid to substitute for `Tm`
-6. If only one `Mi` remains, that method is used for the following rewrites. Otherwise, we instead move to [await any other type].
+      2. The generic parameter of `E` is `Te`
+      3. `Ti` satisfies any constraints on `Tm`
+      4. `Mie` is `Mi` with `Te` substituted for `Tm`, and `Pe` is the resulting parameter of `Mie`
+      5. There is an identity or implicit reference conversion from `E` to the type of `Pe`
+5. If only one `Mi` remains, that method is used for the following rewrites. Otherwise, we instead move to [await any other type].
 
 We'll generally rewrite `await expr` into `System.Runtime.CompilerServices.AsyncHelpers.Await(expr)`. A number of different example scenarios for this are covered below. The
 main interesting deviations are when `struct` rvalues need to be hoisted across an `await`, and exception handling rewriting.

src/Compilers/CSharp/Portable/Binder/Binder.ValueChecks.cs

@@ -4708,6 +4708,10 @@ internal SafeContext GetValEscape(BoundExpression expr, SafeContext localScopeDe
                     // only possible in error cases (if possible at all)
                     return localScopeDepth;
 
+                case BoundKind.ArgList:
+                    // Only possible in error scenarios in runtime async (arglist operators are disallowed in runtime async methods)
+                    return localScopeDepth;
+
                 case BoundKind.ConvertedSwitchExpression:
                 case BoundKind.UnconvertedSwitchExpression:
                     var switchExpr = (BoundSwitchExpression)expr;

src/Compilers/CSharp/Portable/Binder/Binder_Await.cs

@@ -265,6 +265,12 @@ private BoundExpression BindArgListOperator(InvocationExpressionSyntax node, Bin
         {
             bool hasErrors = analyzedArguments.HasErrors;
 
+            if (IsInAsyncMethod() && Compilation.IsRuntimeAsyncEnabledIn(ContainingMemberOrLambda))
+            {
+                // Method '{0}' uses a feature that is not supported by runtime async currently. Opt the method out of runtime async by attributing it with 'System.Runtime.CompilerServices.RuntimeAsyncMethodGenerationAttribute(false)'.
+                diagnostics.Add(ErrorCode.ERR_UnsupportedFeatureInRuntimeAsync, node, ContainingMemberOrLambda);
+            }
+
             // We allow names, oddly enough; M(__arglist(x : 123)) is legal. We just ignore them.
             TypeSymbol objType = GetSpecialType(SpecialType.System_Object, diagnostics, node);
             for (int i = 0; i < analyzedArguments.Arguments.Count; ++i)

src/Compilers/CSharp/Portable/Binder/Binder_Invocation.cs

@@ -1694,20 +1694,20 @@ NamespaceOrTypeOrAliasSymbolWithAnnotations convertToUnboundGenericType()
             }
         }
 
-        internal NamedTypeSymbol GetSpecialType(SpecialType typeId, BindingDiagnosticBag diagnostics, SyntaxNode node)
+        internal NamedTypeSymbol GetSpecialType(ExtendedSpecialType typeId, BindingDiagnosticBag diagnostics, SyntaxNode node)
         {
             return GetSpecialType(this.Compilation, typeId, node, diagnostics);
         }
 
-        internal static NamedTypeSymbol GetSpecialType(CSharpCompilation compilation, SpecialType typeId, SyntaxNode node, BindingDiagnosticBag diagnostics)
+        internal static NamedTypeSymbol GetSpecialType(CSharpCompilation compilation, ExtendedSpecialType typeId, SyntaxNode node, BindingDiagnosticBag diagnostics)
         {
             NamedTypeSymbol typeSymbol = compilation.GetSpecialType(typeId);
             Debug.Assert((object)typeSymbol != null, "Expect an error type if special type isn't found");
             ReportUseSite(typeSymbol, diagnostics, node);
             return typeSymbol;
         }
 
-        internal static NamedTypeSymbol GetSpecialType(CSharpCompilation compilation, SpecialType typeId, Location location, BindingDiagnosticBag diagnostics)
+        internal static NamedTypeSymbol GetSpecialType(CSharpCompilation compilation, ExtendedSpecialType typeId, Location location, BindingDiagnosticBag diagnostics)
         {
             NamedTypeSymbol typeSymbol = compilation.GetSpecialType(typeId);
             Debug.Assert((object)typeSymbol != null, "Expect an error type if special type isn't found");

src/Compilers/CSharp/Portable/Binder/Binder_Symbols.cs

@@ -263,7 +263,7 @@ private BoundForEachStatement BindForEachPartsWorker(BindingDiagnosticBag diagno
                 var placeholder = new BoundAwaitableValuePlaceholder(expr, builder.MoveNextInfo?.Method.ReturnType ?? CreateErrorType());
                 awaitInfo = BindAwaitInfo(placeholder, expr, diagnostics, ref hasErrors);
 
-                if (!hasErrors && awaitInfo.GetResult?.ReturnType.SpecialType != SpecialType.System_Boolean)
+                if (!hasErrors && (awaitInfo.GetResult ?? awaitInfo.RuntimeAsyncAwaitCall?.Method)?.ReturnType.SpecialType != SpecialType.System_Boolean)
                 {
                     diagnostics.Add(ErrorCode.ERR_BadGetAsyncEnumerator, expr.Location, getEnumeratorMethod.ReturnTypeWithAnnotations, getEnumeratorMethod);
                     hasErrors = true;

src/Compilers/CSharp/Portable/Binder/ForEachLoopBinder.cs

@@ -1045,6 +1045,11 @@ private void GetAwaitableInstancePlaceholders(ArrayBuilder<(BoundValuePlaceholde
             {
                 placeholders.Add((placeholder, SafeContextAndLocation.Create(valEscapeScope)));
             }
+
+            if (awaitableInfo.RuntimeAsyncAwaitCallPlaceholder is { } runtimePlaceholder)
+            {
+                placeholders.Add((runtimePlaceholder, SafeContextAndLocation.Create(valEscapeScope)));
+            }
         }
 
         public override BoundNode? VisitImplicitIndexerAccess(BoundImplicitIndexerAccess node)

src/Compilers/CSharp/Portable/Binder/RefSafetyAnalysis.cs

@@ -150,7 +150,7 @@ internal static BoundStatement BindUsingStatementOrDeclarationFromParts(SyntaxNo
 
                 if (awaitableTypeOpt is null)
                 {
-                    awaitOpt = new BoundAwaitableInfo(syntax, awaitableInstancePlaceholder: null, isDynamic: true, getAwaiter: null, isCompleted: null, getResult: null) { WasCompilerGenerated = true };
+                    awaitOpt = new BoundAwaitableInfo(syntax, awaitableInstancePlaceholder: null, isDynamic: true, getAwaiter: null, isCompleted: null, getResult: null, runtimeAsyncAwaitCall: null, runtimeAsyncAwaitCallPlaceholder: null) { WasCompilerGenerated = true };
                 }
                 else
                 {

src/Compilers/CSharp/Portable/Binder/UsingStatementBinder.cs

@@ -0,0 +1,41 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+using System.Diagnostics;
+
+namespace Microsoft.CodeAnalysis.CSharp;
+
+partial class BoundAwaitableInfo
+{
+    private partial void Validate()
+    {
+        if (RuntimeAsyncAwaitCall is not null)
+        {
+            Debug.Assert(RuntimeAsyncAwaitCall.Method.ContainingType.ExtendedSpecialType == InternalSpecialType.System_Runtime_CompilerServices_AsyncHelpers);
+            Debug.Assert(RuntimeAsyncAwaitCallPlaceholder is not null);
+
+            switch (RuntimeAsyncAwaitCall.Method.Name)
+            {
+                case "Await":
+                    Debug.Assert(GetAwaiter is null);
+                    Debug.Assert(IsCompleted is null);
+                    Debug.Assert(GetResult is null);
+                    break;
+
+                case "AwaitAwaiter":
+                case "UnsafeAwaitAwaiter":
+                    Debug.Assert(GetAwaiter is not null);
+                    Debug.Assert(IsCompleted is not null);
+                    Debug.Assert(GetResult is not null);
+                    break;
+
+                default:
+                    Debug.Fail($"Unexpected RuntimeAsyncAwaitCall: {RuntimeAsyncAwaitCall.Method.Name}");
+                    break;
+            }
+        }
+
+        Debug.Assert(GetAwaiter is not null || RuntimeAsyncAwaitCall is not null || IsDynamic || HasErrors);
+    }
+}

src/Compilers/CSharp/Portable/BoundTree/BoundAwaitableInfo.cs

@@ -695,13 +695,19 @@
     <Field Name="Expression" Type="BoundExpression"/>
   </Node>
 
-  <Node Name="BoundAwaitableInfo" Base="BoundNode">
+  <Node Name="BoundAwaitableInfo" Base="BoundNode" HasValidate="true">
     <!-- Used to refer to the awaitable expression in GetAwaiter -->
     <Field Name="AwaitableInstancePlaceholder" Type="BoundAwaitableValuePlaceholder?" Null="allow" />
     <Field Name="IsDynamic" Type="bool"/>
     <Field Name="GetAwaiter" Type="BoundExpression?" Null="allow"/>
     <Field Name="IsCompleted" Type="PropertySymbol?" Null="allow"/>
     <Field Name="GetResult" Type="MethodSymbol?" Null="allow"/>
+    <!-- Refers to the runtime async helper call we use for awaiting. Either this is an instance of a BoundCall of an AsyncHelpers.Await method, and 
+         GetAwaiter, IsCompleted, and GetResult are null, or this is an instance of a AsyncHelpers.AwaitAwaiter/UnsafeAwaitAwaiter BoundCall, and the other
+         fields are not null. -->
+    <Field Name="RuntimeAsyncAwaitCall" Type="BoundCall?" Null="allow"/>
+    <!-- Never null when RuntimeAsyncAwaitCall is not null -->
+    <Field Name="RuntimeAsyncAwaitCallPlaceholder" Type="BoundAwaitableValuePlaceholder?" Null="allow"/>
   </Node>
 
   <Node Name="BoundAwaitExpression" Base="BoundExpression">