static enough metaprogramming

[mraleph]

tldr: I propose we follow the lead of D, Zig and C++26 when it comes to metaprogramming. We introduce an optional (toolchain) feature to force compile time execution of certain constructs. We add library functions to introspect program structure which are required to execute in compile time toolchain supports that. These two together should give enough expressive power to solve a wide range of problems where metaprogramming is currently wanted. cc @dart-lang/language-team

History of dart:mirrors

In the first days of 2017 I have written a blog post "The fear of dart:mirrors" which contained started with the following paragraph:

dart:mirrors might be the most misunderstood, mistreated, neglected component of the Dart's core libraries. It has been a part of Dart language since the very beginning and is still surrounded by the fog of uncertainty and marked as Status: Unstable in the documentation - even though APIs have not changed for a very long time.

In 2017 type system was still optional, AOT was a glorified "ahead-off-time-JIT", and the team maintained at least 3 different Dart front-ends (VM, dart2js and analyzer). Things really started shifting with Dart 2 release: it had replaced optional types with a static type system and introduced common front-end (CFE) infrastructure to be shared by all backends. Dart 3 introduced null-safety by default (NNBD).

And so 8 years and many stable releases later Dart language and its toolchains have changed in major ways, but dart:mirrors remained in the same sorrowful state: a core library only supported by the native implementation on Dart, only in JIT mode and only outside of Flutter.

How did this happen?

The root of the answer lies in the conflict between Dart 1 design philosophy and necessity to use AOT-compilation for deployment.

Dart 1 was all in on dynamic typing. You don't know what a variable contains - but you can do anything with it. Can pass it anywhere. Can all any methods on it. Any class can override a catch-all noSuchMethod and intercept invocations of methods it does not define. Dart's reflection system dart:mirrors is similarly unrestricted: you can reflect on any value, then ask information about its type, ask type about its declarations, ask declared members about their parameters and so on. Having an appropriate mirror you can invoke methods, read and write fields, instantiate new objects.

This ability to indirectly act on the state of the program creates a problem for static analysis of the program and that in turn affects ability of the AOT compiler to produce a small and fast binary.

To put this complexity in simple terms consider two pieces of code:

// Direct
T0 e0; /* ... */; Tn en;
e0.method(e1, /* ... */, en);

// Reflective
InstanceMirror m; List args; Symbol name;
m.invoke(name, args);

When compiler sees the first piece of code it can easily figure out which method implementation this call can reach and what kind of parameters are passed through. With the second piece of code analysis complexity skyrockets - none of the information is directly available in the source code: to know anything about the invocation compiler needs to know a lot about contents of m, args and name.

While it is not impossible to built static analysis which is capable to see through the reflective access - in practice such analyses are complicated, slow and suffer from precision issues on real world code.

AOT compilation and reflection is pulling into opposite directions: AOT compiler wants to know which parts of the program are accessed and how, while reflection obscures this information and provides developer with indirect access to the whole program. When trying to resolve this conflict you can choose between three options:

• The first option is to make reflection system just work even after AOT compilation. This means retaining all information (and code) which can be indirectly accessed via reflection. In practice this means retaining most of the original program - because most uses of reflection are notoriously hard to analyze statically.
• The second option is to allow AOT-compiler to ignore reflection uses it can't analyze and providing developer with a way to feed additional information about reflective uses into the compiler. If developer forgets (or feeds incomplete or incorrect information) reflective code might break after compilation if compiler decides to remove part of the program which it deems unreachable.
• The third option is to capitulate and disable reflection APIs in AOT compiled code.

Facing this choice is not unique to Dart: Java faces exactly the same challenge. On one hand, the package java.lang.reflect provides indirect APIs for accessing and modifying the state and structure of the running program. On the other hand, developers want to obfuscate and shrink their apps before deployment. Java ecosystem went with the second option: shrinking tools more-or-less ignore reflection and developers have to manually inform toolchain about the program elements which are accessed reflectively.

Note

There has been a number of attempts to statically analyze reflection in Java projects, but they have all hit issues around scalability and precision of the analysis. See:

• Reflection Analysis for Java
• Understanding and Analyzing Java Reflection
• Challenges for Static Analysis of Java Reflection – Literature Review and Empirical Study

Graal VM Native Image (AOT compiler for Java) attempts to fold away as much of reflection uses as it can, but otherwise just like ProGuard and similar tools relies on the developer to inform compiler about reflection uses it could not resolve statically.

R8 (Android bytecode shrinker) has a special troubleshooting section in its README to cover obscure situations which might arise if developer fails to properly configure ProGuard rules to cover reflection uses.

Reflekt: a Library for Compile-Time Reflection in Kotlin describes a compiler plugin based compile time reflection system similar in some ways to reflectable.

Compile-time Reflection and Metaprogramming for Java covers a metaprograming system which proposes metaprogramming system based on compile-time reflection.

Dart initially went with the first option and tried to make dart:mirrors just work when compiling Dart to JavaScript. However, rather quickly dart2js team started facing performance and code size issues caused by dart:mirrors in large Web applications. So they switched gears and tried the second option: introduced @MirrorsUsed annotation. However it provided only a temporary and partial reprieve from the problems and was eventually abandoned together with dart:mirrors.

There were two other attempts to address code size issues caused by mirrors, while retaining some amount of reflective capabilities: now abandoned package smoke and still maintained package reflectable. Both of these apply similar approach: instead of relying on the toolchain to provide unrestricted reflection, have developer opt-in into specific reflective capabilities for specific parts of the program then generate a pile of auxiliary Dart code implementing these capabilities.

Note

Another exploration similar in nature was (go/const-tree-shakeable-reflection-objects)[http://go/const-tree-shakeable-reflection-objects].

Fundamentally both of these approaches were dead ends and Web applications written in Dart solved their code size and performance issues by moving away from reflection to code generation, effectively abandoning runtime metaprogramming in favor of build time metaprogramming. Code generators are usually written on top of Dart's analyzer package: they inspect (possibly incomplete) program structure and produce additional code which needs to be compiled together with the program.

Following this experience, we have decided to completely disable dart:mirrors when implementing native AOT compiler.

Note

For the sake of brevity I am ignoring discussion of performance problems associated with reflection for now. It is sufficient to say that naive implementation of reflection is guaranteed to be slow and minimizing the cost likely requires runtime code generation - which is not possible in all environments.

Note

If you are familiar with intricacies of Dart VM / Flutter engine embedding you might know that Dart VM C API is largely reflective in nature: it allows you to look up libraries, classes and members by their names. It allows you invoke methods and set fields indirectly. That why @pragma('vm:entry-point') exists - and that is why you are required to place it on entities which are accessed from outside of Dart.

const

Let me change gears for a moment and discuss Dart's const and its limitations. This feature gives you just enough power at compile time to:

• construct objects (via const constructors),
• create constant list and map literals,
• perform arithmetic on int and double values
• perform logical operations on bool values
• compare primitive values
• ask length of a constant String

Exhaustive list is given in section 17.3 of Dart Programming Language Specification and even though the description occupies 5 pages the sublanguage it defines is very small and excludes a lot of expressions which feel like they should actually be included. It just feels wrong that const x = [].length is invalid while const x = "".length is valid. For some seemingly arbitrary reason String.length is the only blessed property which can't be accessed in a constant expression. You can't write [for (var i = 0; i < 10; i++) i] and so on.

Consider the following code from dart:convert internals:

static const int CHAR_SIMPLE_STRING_END = 1;
static const int CHAR_WHITESPACE = 2;

/**
 * [_characterAttributes] string was generated using the following code:
 *
 * ```
 * int $(String ch) => ch.codeUnitAt(0);
 * final list = Uint8List(256);
 * for (var i = 0; i < $(' '); i++) {
 *   list[i] |= CHAR_SIMPLE_STRING_END;
 * }
 * list[$('"')] |= CHAR_SIMPLE_STRING_END;
 * list[$('\\')] |= CHAR_SIMPLE_STRING_END;
 * list[$(' ')] |= CHAR_WHITESPACE;
 * list[$('\r')] |= CHAR_WHITESPACE;
 * list[$('\n')] |= CHAR_WHITESPACE;
 * list[$('\t')] |= CHAR_WHITESPACE;
 * for (var i = 0; i < 256; i += 64) {
 *   print("'${String.fromCharCodes([
 *         for (var v in list.skip(i).take(64)) v + $(' '),
 *       ])}'");
 * }
 * ```
 */
static const String _characterAttributes =
    '!!!!!!!!!##!!#!!!!!!!!!!!!!!!!!!" !                             '
    '                            !                                   '
    '                                                                '
    '                                                                ';

It feels strangely limiting that the only way to update this constant is to modify the comment above it, copy that comment into a temporary file, run it and paste the output back into the source. What we really want is to define _characterAttributes in the following way:

static const int CHAR_SIMPLE_STRING_END = 1;
static const int CHAR_WHITESPACE = 2;

static const String _characterAttributes = _computeCharacterAttributes();

static String _computeCharacterAttributes() {
  int $(String ch) => ch.codeUnitAt(0);
  final list = Uint8List(256);
  for (var i = 0; i < $(' '); i++) {
    list[i] |= CHAR_SIMPLE_STRING_END;
  }
  list[$('"')] |= CHAR_SIMPLE_STRING_END;
  list[$('\\')] |= CHAR_SIMPLE_STRING_END;
  list[$(' ')] |= CHAR_WHITESPACE;
  list[$('\r')] |= CHAR_WHITESPACE;
  list[$('\n')] |= CHAR_WHITESPACE;
  list[$('\t')] |= CHAR_WHITESPACE;
  return String.fromCharCodes(list);
}

This requires the definition of constant expression to be expanded to cover a significantly larger subset of Dart than it currently includes. Such feature does however exist in other programming languages, most notably C++, D, and Zig.

C++

Originally metaprogramming facilities provided by C++ were limited to preprocessor macros and template metaprogramming. However, C++11 added constexpr and C++20 added consteval.

The following code is valid in modern C++ and computes kCharacterAttributes table in compile time.

constexpr uint8_t CHAR_SIMPLE_STRING_END = 1;
constexpr uint8_t CHAR_WHITESPACE = 2;

constexpr auto kCharacterAttributes = []() {
  std::array<uint8_t, 256> list {};
  for (int i = 0; i < ' '; i++) {
    list[i] |= CHAR_SIMPLE_STRING_END;
  }
  list['"'] |= CHAR_SIMPLE_STRING_END;
  list['\\'] |= CHAR_SIMPLE_STRING_END;
  list[' '] |= CHAR_WHITESPACE;
  list['\r'] |= CHAR_WHITESPACE;
  list['\n'] |= CHAR_WHITESPACE;
  list['\t'] |= CHAR_WHITESPACE;
  return list;
}();

Note

C++26 will most likely include reflection support which would allow the program to introspect and modify its structure in compile time. Reflection would allow programmer achieve results similar to those described in the next section about D. I am omitting it from discussion here because it is not part of the language just yet.

D

C++ example given above can be trivially translated to D, which also supports compile time function execution (CTFE).

static immutable CHAR_SIMPLE_STRING_END = 1;
static immutable CHAR_WHITESPACE = 2;

static immutable ubyte[256] CharacterAttributes = () {
    ubyte[256] list;
    for (int i = 0; i < ' '; i++) {
      list[i] |= CHAR_SIMPLE_STRING_END;
    }
    list['"'] |= CHAR_SIMPLE_STRING_END;
    list['\\'] |= CHAR_SIMPLE_STRING_END;
    list[' '] |= CHAR_WHITESPACE;
    list['\r'] |= CHAR_WHITESPACE;
    list['\n'] |= CHAR_WHITESPACE;
    list['\t'] |= CHAR_WHITESPACE;
    return list;
}();

D however takes this further: it provides developer means to introspect and modify the structure of the program itself in compile time. Introspection is achieved via traits and modifications are possible via templates and template mixins.

Consider the following example which defines a template function fmt capable of formatting arbitrary structs:

string fmt(T)(T o)
{
    // T.stringof will return the name of a type
    string result = T.stringof ~ " { ";
    bool comma = false;
    
//  This foreach loop is expanded in compile time by copying
//  the body of the loop for each element of the aggregate 
//. and substituting memberName with corresponding constant.
//  vvvvvvvvvvvvvv 
    static foreach (memberName; [__traits(allMembers, T)])
    //                                    ^^^^^^^^^^
    // Trait allMembers returns names of all members of T
    // as sequence of string literals.
    {
        if (comma)
            result ~= ", ";
        result ~= memberName ~ ": "
        result ~= fmt(__traits(getMember, o, memberName));
        //                     ^^^^^^^^^
        // Trait getMember allows to construct member access 
        // expression o.memberName - memberName has to be 
        // a compile time constant string.
        comma = true;
    }
    result ~= "}";
    return result;
}

string fmt()(int o)
{
    return format("%d", o);
}

string fmt()(string o)
{
    return o;
}

struct Person {
  string name;
  int age;
}

write(fmt(Person("Nobody", 42))); // Person { name: Nobody, age: 42 }

When you instantiate fmt!Person compiler effectively produces the following code

// Specialization of fmt for a Person.
string fmt!Person(Person o)
{
    // T.stringof will return the name of a type
    string result = "Person" ~ " { ";
    bool comma = false;    
    {
        if (comma)
            result ~= ", ";
        result ~= "name" ~ ": "
        result ~= fmt(o.name);
        comma = true;
    }
    {
        if (comma)
            result ~= ", ";
        result ~= "age" ~ ": "
        result ~= fmt(o.age);
        comma = true;
    }
    result ~= "}";
    return result;
}

See Compile-time vs. compile-time for an introduction into D's compile-time metaprogramming.

Zig

Zig metaprogramming facilities are centered around comptime - a modifier which requires variable to be known at compile-time. Zig elevates types to be first-class values, meaning that you can put type into a variable or write a function which transforms one type into another type, but requires that types are only used in expressions which can be evaluated in compile-time.

While Zig's approach to types is fairly unique, the core of its metaprogramming facilities is strikingly similar to D:

• A number of builtin functions are provided which allow program to interact with the compiler as it is being compiled. For example, Zig's std.meta.fields(@TypeOf(o)) is equivalent of D's __traits(allMembers, T), while @field(o, name) is equivalent of __traits(getMember, o, name).
• A number of constructs are provided to facilitate compile-time specialization, e.g. Zig's inline for is expanded in compile time just like D's static foreach.

Here is an example which implements a generic function print, similar to generic fmt we have implemented above:

const std = @import("std");
const builtin = @import("builtin");

// anytype is a placeholder for a type, asking compiler to 
// infer type at callsite.
//              vvvvvvv
pub fn print(o: anytype) void {
    const t: type = @TypeOf(o);
    //              ^^^^^^^
    // @TypeOf is a builtin function which returns type 
    // of an expression.

    
    // Types are values so you can just switch over them
    switch (t) {
        // Handle u8 and u8 slices
        u8 => std.debug.print("{}", .{o}),
        []const u8, []u8 => std.debug.print("{s}", .{o}),
        
        // Handle everything else 
        else => switch (@typeInfo(t)) {
            .Struct => |info| {
                // @typeName provides name of the given type
                std.debug.print("{s} {{ ", .{@typeName(t)});
                var comma = false;

                // inline loops are expanded in compile time.
				inline for (info.fields) |field| {
                    if (comma) {
                        std.debug.print(", ", .{});
                    }
                    std.debug.print("{s}: ", .{field.name});
                    // @field allows to access a field by name known
                    // at compile time. @as performs a cast.
                    print(@as(field.type, @field(o, field.name)));
                    comma = true;
                }
                std.debug.print("}}", .{});
            },
            else => @compileError("Unable to format " ++ @typeName(t)),
        },
    }
}

const Person = struct {
    name: []const u8,
    age: u8,
};

pub fn main() !void {
    print(Person{ .name = "Nobody", .age = 42 });
}

Dart and Platform-specific code

Dart's does not have a powerful compile time execution mechanism similar to those described above. Or does it?

Consider the following chunk of code which one could write in their Flutter application:

static Widget get _buttonText => switch (defaultTargetPlatform) {
    TargetPlatform.android => AndroidSpecificWidget(),
    TargetPlatform.iOS => IOSSpecificWidget(),
    TargetPlatform.fuchsia => throw UnimplementedError(),
  };

Developer compiling their application for Android would naturally expect that the final build only contains AndroidSpecificWidget() and not IOSSpecificWidget() and vice versa. This expectation is facing one challenge: defaultTargetPlatform is not a simple constant - it is defined as result of a computation. Here is its definition from Flutter internals:

platform.TargetPlatform get defaultTargetPlatform {
  platform.TargetPlatform? result;
  if (Platform.isAndroid) {
    result = platform.TargetPlatform.android;
  } else if (Platform.isIOS) {
    result = platform.TargetPlatform.iOS;
  } else if (Platform.isFuchsia) {
    result = platform.TargetPlatform.fuchsia;
  } else if (Platform.isLinux) {
    result = platform.TargetPlatform.linux;
  } else if (Platform.isMacOS) {
    result = platform.TargetPlatform.macOS;
  } else if (Platform.isWindows) {
    result = platform.TargetPlatform.windows;
  }
  assert(() {
    if (Platform.environment.containsKey('FLUTTER_TEST')) {
      result = platform.TargetPlatform.android;
    }
    return true;
  }());
  if (kDebugMode && platform.debugDefaultTargetPlatformOverride != null) {
    result = platform.debugDefaultTargetPlatformOverride;
  }
  if (result == null) {
    throw FlutterError(
      'Unknown platform.\n'
      '${Platform.operatingSystem} was not recognized as a target platform. '
      'Consider updating the list of TargetPlatforms to include this platform.',
    );
  }
  return result!;
}

None of Platform.isX values are const's either: they are all getters on the Platform class.

This seems rather wasteful: even though AOT compiler knows precisely which platform it targets developer has no way of writing their code in a way that is guaranteed to be tree-shaken based on this information. At least not within the language itself - last year we have introduced support for two @pragmas: vm:platform-const-if and vm:platform-const which allow developer to inform the compiler that a function can and should be evaluated at compile time if compiler knows the platform it targets.

These annotations were placed on all API surfaces in Dart and Flutter SDK which are supposed to evaluate to constant when performing release builds:

// Dart SDK
abstract final class Platform {
  @pragma("vm:platform-const")
  static final pathSeparator = _Platform.pathSeparator;
  // ...
  @pragma("vm:platform-const")
  static final operatingSystem = _Platform.operatingSystem;

  @pragma("vm:platform-const")
  static final bool isLinux = (operatingSystem == "linux");

  // ...
}

// Flutter SDK
@pragma('vm:platform-const-if', !kDebugMode)
platform.TargetPlatform get defaultTargetPlatform {
  // ...
}

An implementation of this feature leans heavily on an earlier implementation of const-functions experiment. This experiment never shipped as a real language feature, but CFE's implementation of constant evaluation was expanded to support significantly larger subset of Dart than specification currently permits for const expressions, including imperative loops, if-statements, List and Map operations.

Static Enough Metaprogramming for Dart

Let us first recap History of dart:mirrors: reflection posed challenges for Dart because it often makes code impossible to analyze statically. The ability to analyze the program statically is crucial for AOT compilation, which is the main deployment mode for Dart. Dart answer to this was to shift metaprogramming from run time to (pre)build time by requiring code generation: an incomplete program structure can be inspected via analyzer package and additional code can be generated to complete the program. This way AOT compilers see a static program structure and don't need to retain any reflective information.

To put it simply, we avoid reflection because our AOT compilers can't analyze it and fold it away. Conversely, if compiler could analyze and fold reflection away we would not need to avoid it. Dart could have its cake and eat it too. D, Zig (and C++26) show us the path: we need to lean on compile time constant evaluation to achieve that.

I propose we introduce a special metadata constant konst in the dart:metaprogramming which would allow developer to request enhanced constant evaluation at compile time if the underlying compiler supports it.

/// Forces Dart compiler which supports enhanced constant evaluation to
/// compute the value of the annotated variable at compile time.
const konst = pragma('konst');

Applying @konst to normal variables and fields simply requests compiler to compute their value at compile time:

@konst
static final String _characterAttributes = _computeCharacterAttributes();
// _computeCharacterAttributes() is evaluated at compile time if compiler
// supports it.

When @konst is applied to parameters (including type parameters) it turns functions into templates: compiler will require that annotated parameter is a constant known at compile time and clone the function for a specific combination of parameters. The original function is removed from the program: it is impossible to invoke it dynamically or tear it off. To annotate this as @konst developer will need to place @konst on the declaration of the function itself.

Important

Here and below we assume that constant evaluator supports execution of functions (i.e. as implemented by const-functions language experiment) - rather than just a limited subset of Dart required by the language specification. This means [1].first and even [1].map((v) => v + 1).first can be folded to a constant when used in @konst-context.

class X {
  final int v;
  const X(this.v);

  @konst
  String fmt() => '$v'; 
}

void foo<@konst T>(@konst T value) {
  // ...
}

void bar(@konst String v) {
  foo(v);  // ok: T is String and v is @konst itself
}

foo(1);  // ok: T is int, value is 1
foo([1].first);  // OK: T is int, value is 1
bar('a'); // ok
const X(1).fmt(); // ok
X(1).fmt(); // ok

void baz(String v, X x) {
  foo(v);  // error: v is not a constant
  x.fmt();  // error: x is not a constant
}

When @konst is applied to loop iteration variables it instructs the compiler to expand the loop at compile time by first computing the sequence of values for that iteration variable, then cloning the body for each value in order and substituting iteration variable with the corresponding constant.

void foo(@konst int v) {
  // 
} 

for (@konst final v in [1, 2, 3]) {
  foo(v);
}
// expands to: foo(1); foo(2); foo(3);

Generics introduce an interesting caveat though:

void bar<@konst T>(@konst T v) {

}

for (@konst final v in [1, '2', [3]]) {
  bar(v);
}
// expands to: bar<Object>(1); bar<Object>('2'); bar<Object>([3]);
// but what if developer wants specialization to a type?

We could expand dart:metaprogramming with a typeOf(...) helper:

/// Returns the type of the given object. 
///
/// Note: similar to [Object.runtimeType] but will error if v is not a constant
/// and we are running in environment which supports @konst.
external Type typeOf(@konst Object? v);

But that does not solve the problem. Type arguments and normal values are separated in Dart - which means you can't invoke a generic function with the given Type value as type argument, even if Type value is a compile time constant. To breach this boundary we need a helper which would allow us to constructing function invocations during compile time execution.

For example:

external T invoke<T>(@konst Function f, List positional, {
  Map<String, Object?> named = const {},
  List<Type> types = const [],
});

Note

Function.apply does not support passing type arguments to functions, but even if it did we would not want to use it here because we want to enforce compile time expansion of invoke(...) into a corresponding call or an error, if such expansion it not possible.

Combining typeOf and invoke yields expected result:

void bar<@konst T>(@konst T v) { }

for (@konst final v in [1, '2', [3]]) {
  invoke(bar, [v], types: [typeOf(v)]);
}
// expands to: bar<int>(1); bar<String>('2'); bar<List<int>>(3);

You might notice that invoke is a bit wonky: f is @konst, but neither position, nor named, nor types are. Why is that? Well, that's because invoke tries to capture expressivity of a normal function call site: each call site has constant shape (e.g. known number of positional and type arguments, known names for named arguments), but actual arguments are not required to be constant. Dart's type system does not provide good tools to express this, List and Map don't have their shape (e.g. length or keys) as part of their type.

This unfortunately means that compiler needs to be capable of figuring out the shape of lists and maps that flow into invoke. Consider for example that we might want to construct argument sequence imperatively:

void invokeBar(Map<String, Object> values) {
  final named = <String, Object?>{};
  for (@konst final k in ['a', 'b']) {
    named[k] = values[k];
  }
  invoke(bar, [], named: named); // expands to bar(a: input['a'], b: input['b'])
}

Should this code compile? Maybe we could limit ourselves to supporting only collection literals as arguments to invoke:

void invokeBar(Map<String, Object> values) {
  invoke(bar, [], named: {
    for (@konst final k in ['a', 'b']) 
      k: input['k'],
  });
  // expands to bar(a: input['a'], b: input['b'])
}

@konst reflection

Features described above lay the foundation of compile time metaprogramming, but for it to be complete we need to expose more information about the structure of the program.

For example (these are not exhaustive or exact):

// dart:metaprogramming

final class TypeInfo<T> {
  const TypeInfo._();

  /// Obtain `TypeInfo` for the given type `T`.
  external static TypeInfo<T> of<@konst T>();

  /// Is `T` nullable?
  @konst external bool isNullable;

  /// Erase nullability of `T` if it is nullable.
  @konst external TypeInfo<T> get nonNullable;

  /// Return underlying type `T`.
  @konst external Type type;

  /// Check if `T` is subtype of `Base`.
  @konst external bool isSubtypeOf<@konst Base>();

  /// Find instantiation of `Base` in supertypes
  /// of `T` and return the corresponding `TypeInfo`.
  @konst external TypeInfo<Base>? instantiationOf<@konst Base>();

  /// Return type-arguments of `T` if any.	
  @konst external List<TypeInfo> get typeArguments;

  /// Return `T` default constructor.
  @konst external Function defaultConstructor;

  /// Return the list of fields in `T`.
  @konst external List<FieldInfo<T, Object?>> get fields;
}

/// Information about the field of type [FieldType] in the
/// object of type [HostType].
final class FieldInfo<HostType, FieldType> {
  const FieldInfo._();

  @konst external String get name;
 
  @konst external bool get isStatic;

  @konst external TypeInfo<FieldType> get type;

  /// Get the value of this field from the given object. 
  @konst external FieldType getFrom(HostType value);
}

Note that all methods are annotated with @konst so if compiler supports @konst these must be invoked on constant objects and will be folded away - compiler does not need to store any information itself.

It's a spectrum of choice

I have intentionally avoided saying that @konst has to be a language feature and that any Dart implementation needs to support compile time constant evaluation of @konst. I think we should consider doing this as a toolchain feature, similar to how platform-const is implemented.

For example, a native JIT or DDC (development mode JS compiler) could simply implement TypeInfo on top of runtime reflection. This way developer can debug their reflective code as if it was any other Dart code. A deployment compiler (native, Wasm or JS) can then fold the cost of reflection away by enforcing const-ness requirements implied by @konst and folding away reflective operations.

Note

A deployment compiler can even choose between producing specialized code by cloning and specializing functions with @konst-parameters or it could choose to retain reflective metadata and forego cloning at the cost of runtime performance. This reduces the size of deployed applications but decreases peak performance.

In this model, developer might encounter compile time errors when building release application which they did not observe while developing - as development and deployment toolchains implement different semantics.

I think that's an acceptable price to pay for the convenience&power of this feature. We can later choose to implement additional checks in development toolchains or analyzer to minimize amount of errors which are only surfaced by release builds. But I don't see this as a requirement for shipping this feature.

Prototype implementation

To get the feeling of expressive power, implementation complexity and costs I have thrown together a very rough prototype implementation which can be found here. When comparing manual toJSON implementation with a similar (but not equivalent!) one based on @konst reflection I got the following numbers:

• Average code size overhead per class: 270 bytes
• Average JIT kernel generation overhead per-class: 0.2ms (cost of producing specialized functions using Kernel-to-Kernel AST transformation)
• Average AOT compilation overhead per-class: 2.6ms

I think the main difference between manual and reflective implementations is handling of nullable types and lists. Manual implementation inlined both - while reflective leaned on having helper methods for these. I will take a closer look at this an update this section accordingly.

Example: Synthesizing JSON serialization

Note

These are toy examples to illustrate the capabilities rather than full fledged competitor to json_serializable. I have written this code to experiment with the prototype implementation which I have concocted in a very limited time frame.

toJson<@konst T>

Map<String, Object?> toJson<@konst T>(T value) => {
    for (@konst final field in TypeInfo.of<T>().fields)
      if (!field.isStatic) field.name: field.getFrom(value),
  };

// Example
class A {
  final int a;
  final String b;
  A({required this.a, required this.b});
}

// Calling toJson<A>(A(...)) produces specialization
Map<...> toJson$A(A value) => {a: value.a, b: value.b};

fromJson<@konst T>

T fromJson<@konst T>(Map<String, Object?> json) {
  final typeInfo = TypeInfo.of<T>();
  return invoke<T>(
        typeMirror.defaultConstructor,
        [],
        named: {
          for (@konst final field in typeInfo.fields)
            if (!field.isStatic)
              field.name: invoke(
                _valueFromJson,
                [json[field.name]],
                types: [field.type.type],
              ),
        },
      );
}

FieldType _valueFromJson<@konst FieldType>(Object? value) {
  var fieldType = TypeInfo.of<FieldType>();
  if (fieldType.isNullable) {
    if (value == null) {
      return null as FieldType;
    }
    fieldType = fieldType.nonNullable;
  } else {
    if (value == null) {
      throw ArgumentError('Field not found in incoming json');
    }
  }

  // Primitive values are mapped directly.
  if (fieldType.isSubtypeOf<String>() ||
      fieldType.isSubtypeOf<num>() ||
      fieldType.isSubtypeOf<bool>()) {
    return value as FieldType;
  }

  // Lists are unpacked element by element.
  if (fieldType.instantiationOf<List>() case final instantiation?) {
    final elementType = instantiation.typeArguments.first.type;
    return invoke<FieldType>(
          listFromJson,
          [value as List<Object?>],
          typeArguments: [elementType],
        );
  } else {
    // We assume that this is Map -> class conversion then.
    return fromJson<FieldType>(value as Map<String, Object?>);
  }
}

List<E> _listFromJson<@konst E>(List<Object?> list) {
  return <E>[for (var v in list) _valueFromJson<E>(v)];
}

// Example
class A {
  final int a;
  final String b;
  A({required this.a, required this.b});
}

// Calling fromJson<A>({...}) produces specializations:

A fromJson$A(Map<String, Object?> map) {
  return A(
    a: _valueFromJson$int(map['a']),
    b: _valueFromJson$String(map['a']),
  );
}

int _valueFromJson$int(Object? value) {
  if (value == null) {
    throw ArgumentError('Field not found in incoming json');
  }
  return value as int;
}

String _valueFromJson$String(Object? value) {
  if (value == null) {
    throw ArgumentError('Field not found in incoming json');
  }
  return value as String;
}

Example: Defining hashCode and ==

We could also instruct compiler to handle mixin's (and possibly all generic classes) with @konst type parameters in a special way: clone their declarations with known type arguments. This would allow to write the following code:

mixin DataClass<@konst T> {
  @override
  operator ==(Object? other) {
    if (other is! T) {
      return false;
    }

    final typeInfo = TypeInfo.of<T>();
    for (@konst final field in typeInfo.fields) {
      final value1 = field.getFrom(this as T);
      final value2 = field.getFrom(other);
      if (field.type.isSubtypeOf<List>()) {
        if ((value1 as List).length != (value2 as List).length) {
          return false;
        }
        for (var i = 0; i < value1.length; i++) {
          if (value1[i] != value2[i]) {
            return false;
          }
        }
      } else if (value1 != value2) {
        return false;
      }
    }
    return true;
  }

  @override
  int get hashCode {
    final typeInfo = TypeInfo.of<T>();
    var hash = HashHelpers._seed;
    for (@konst final field in typeInfo.fields) {
      hash = HashHelpers.combine(hash, field.getFrom(this as T).hashCode);
    }
    return HashHelpers.finish(hash);
  }

  Map<String, Object?> toJson() => toJsonImpl<T>(this as T);
}

// Example
class A with DataClass<A> {
  final int a;
  final String b;
  // ...
}

// Bodies of members in DataClass<A> are copied and specialized for a known
// value of T. This means A automatically gets definitions of operator== and
// get hashCode in terms of its fields.

References

[rrousselGit]

I wonder what the performance overhead would be compared to what you could achieve using code-generation.

Is it safe to assume that raw Dart code would still be faster ; due to:

• not having to lookup type information
• the compiler being able to perform the usual optimisations (like how comparing List<int> is faster than comparing List<Object>)

[stereotype441]

At first glance, this sounds like a really cool idea. I'm curious to play around with it more and try to understand its uses and limitations.

To get the feeling of expressive power, implementation complexity and costs I have thrown together a very rough prototype implementation which can be found [here][konst-prototype-branch].

I'm very interested in looking at this! Did you forget to paste in the link?

[lrhn]

(Working my way through the proposal, so I might not have all the details yet.)

The example:

void bar<@konst T>(@konst T v) { }

for (@konst final v in [1, '2', [3]]) {
  invoke(bar, [v], types: [typeOf(v)]);
}

doesn't vibe with the

enhanced constant evaluation at compile time if the underlying compiler supports it.

goal. If the compiler doesn't support konst-evaluation, it would have to be able implement invoke and typeOf as runtime functions. (But I see that you say that: Every compilation tool-chain must support the functionality, the option is between doing it at compile-time and using runtime reflection. And the only thing preventing this from being full-fledged reflection is that every tool-chain will require that @konst-annotated arguments are actually available at compile-time.)

The reason you need the invoke may be that the loop is using the wrong abstraction level function.
It's treated as if it's iterating over values, which is why it has a static type of Object, but since the loop is expanded and the elements of the list is inserted into the body as expressions, it is really looping over expressions, not values.
If the entries are treated as expressions, with no given type, then substituting them into separate unrollings of bar(v)
doesn't have to assume that the static type of v is Object, and then you can get inference for each invocation.

But then it won't be able to have the same semantics if executed at runtime if the compiler doesn't do konst.
If that's a requirement, then this proposal is very limited in what it can do (it probably cannot touch any mutable global state, and can't rely on object identity).

This is a two-level language, like Scheme's quote/unquote, but it's omitting the quote on the list elements and the unquote in the body of the konst loop. That makes it harder to reason about.

(The substitution into loops is also worrisome, take @konst var ctr = 0; for (@konst var i in [++ctr, ++ctr]) { print("${i} == ${i} < $ctr"); }. If the loop is actually unrolled by substituting the expression with a side-effect into the body more than once, then this behaves very differently than if run at runtime. A more viable subsititution would be to convert for (@konst var i in [expr1, ..., exprN]) { body[i]; } into

{
var i$0 = expr1;
var i$1 = expr2;
...
var i$N = exprN;
{
  body[i$0];
}
// ...
{
  body[i$N];
}

What if we had quote and unquote functions defined in dart:metaprogramming, which captures a value and its static type as an Expr<T> which has an R unquote<R>(R Function<X>(X) use); function to unpack itself.
Then you'd write this as:

void bar<@konst T>(@konst T v) { }

for (@konst final v in [quote(1), quote('2'), quote([3])]) {
  v.unquote(<T>(o) => bar<T>(o));
}

Still would't work the same if executed at runtime.

Would be:

@konst
class Expr<T> {
  final T value;
  Expr(@konst this.value);
  @konst
  R unquote<R>(R Function<X>(X value) use) => use(value);
}
Expr<T> quote<T>(T value) => Expr<T>(value);

(Or just call the class Quote.)

[lrhn]

I worry about the field-reflection because it doesn't say what a "field" is.

If it's any getter, then this is probably fine.

If it distinguishes instance variables from getters, then it's almost certainly not fine. That'd be breaking the abstraction of the class, and of its superclasses.

Which also means that if you do:

for (@konst final field in TypeInfo.of<T>().fields) 
      if (!field.isStatic) field.name: field.getFrom(value)

you'll probably also need a && field.name != 'hashCode' && field.name != 'runtimeType' guard.
Which sucks, but so does breaking abstraction.

(Can you see private fields? If you are in the same library?)

[aam]

I have thrown together a very rough prototype implementation which can be found [here][konst-prototype-branch].

Can you fix the link?

[tatumizer]

FWIW: the review would be more complete if you also consider Julia's metaprogramming as a source of ideas.

[TekExplorer]

Wouldn't it be simpler to simply expand access to const, and (less simply) just give access to reflection there and only there?
like:

// figure out how to mark this as constant in all ways.
external T constInvoke<const F extends Function>(F fn, posParams, namedParams, {List<Type> typeArguments}) const;

const Object? toJson<T>(T object) {...}
// could be a modifier like async
Object? toJson<T>(T object) const {
  if ((T).annotations.hasCustomToJson) {
    // ...
  }
  return switch (object) {
    null || String() || num() || bool() => object,
    _ => toJsonMap(object),
  };
}
Map<String, Object?> toJsonMap<T>(T object) const {
  return <String, Object?>{
    // probably break this out more to check for annotations and such.
    for (final field in (T).fields) field.name: constInvoke(toJson, [field.of(object)], typeArguments: [field.type])
  };
}

and possibly have const? for potentially constant vs must be constant.

after all, there are often classes (read: annotations) that arent ever supposed to not be const.

[schultek]

But that does not solve the problem. Type arguments and normal values are separated in Dart - which means you can't invoke a generic function with the given Type value as type argument, even if Type value is a compile time constant.

Wouldn't it be an option to allow exactly that, instead of adding the "wonky" invoke function?

So

void bar<@konst T>(@konst T v) { }

for (@konst final v in [1, '2', [3]]) {
  final t = typeOf(v);
  bar<t>(v);
}

would be allowed since the generic parameter is a compile time constant?

[tatumizer]

void bar<@konst T>(@konst T v) { }

for (@konst final v in [1, '2', [3]]) {
  final t = typeOf(v);
  bar<t>(v);
}

While reading this program, I cannot easily see that the for block executes in comptime. Even if the compiler can somehow figure this out (not clear how), for the reader it looks quite baffling.
It should be

comptime for (final v in [1, '2', [3]]) {
  final t = typeOf(v);
  bar<t>(v);
}
// more general form:
comptime { 
  for (final v in [1, '2', [3]]) {
    final t = typeOf(v);
    bar<t>(v);
  }
  // other stuff executed in comptime
  //...
}

[julemand101]

To get the feeling of expressive power, implementation complexity and costs I have thrown together a very rough prototype implementation which can be found [here][konst-prototype-branch].

I'm very interested in looking at this! Did you forget to paste in the link?

I think the link should have been: https://github.com/mraleph/sdk/tree/static_enough_reflection. At least that branch matches this proposal and have a recent commit related to this topic: mraleph/sdk@d594629

[mraleph]

RE: @rrousselGit

I wonder what the performance overhead would be compared to what you could achieve using code-generation.

The performance overhead at which stage and at which mode? If we are talking about runtime performance for the code compiled with toolchain which supports @konst then hard written (or generated code) and @konst should have very similar performance characteristics. Though the comparison is very nuanced here:

1. On one side, there are no type information lookups or anything like that in the final code. If you use @konst type arguments the final code will be fully specialized for specific types. This is the level of optimization which is not easily achievable in Dart today (sans manual function cloning). In some occasions it can be faster - because it replaces type parameters with concrete types and removes const associated with reified type parameters.

2. On another side, you need to be about levels of abstraction. You might be encouraged to create helpers (e.g. see my example fromJson implementation which defines listFromJson helper). In code written by hand (or generated via a code generator) you will create the same helper but it will most likely be generating a code snippet to be used inline inside bigger chunk of code. This means generated code would have less levels of abstractions for compiler to chew through. It is not unsurmountable - but it is a difference to be aware off.

   // @konst approach
   FieldType _valueFromJson<@konst FieldType>(Object? value) {
     // ...
     if (fieldType.instantiationOf<List>() case final instantiation?) {
       final elementType = instantiation.typeArguments.first.type;
       return invoke<FieldType>(
             listFromJson,
             [value as List<Object?>],
             typeArguments: [elementType],
           );
     }
     // ...
   }
   
   List<E> _listFromJson<@konst E>(List<Object?> list) {
     return <E>[for (var v in list) _valueFromJson<E>(v)];
   }
   
   // Instantiating _valueFromJson<List<A>> yields:
   
   List<A> _valueFromJson$ListA(Object? value) {
     return _listFromJson$A(value as List<Object?>);
   }
   
   A _listFromJson$A(List<Object?> value) {
     return <E>[for (var v in list) _valueFromJson$A(v)];
   }
   
   // Levels of abstractions are still present.

   // Codegeneration approach (pseudocode)
   String genValueFromJson(String varName, Info info) {
     // ...
     if (info.isList) {
       return genListFromJson(varName, info.elementType);
     }
     // ...
   }
   
   String genListFromJson(String varName, Info info) {
     return '<${info.typeName}>[for (var v in list) ${genValueFromJson('v', info)}]';
   }
   
   // Running code generator you will end up with a snippet:
   // (abstractions are kinda automatically erased)
   
   <A>[for (var v in list) A.fromJson(v)]

I'm very interested in looking at this! Did you forget to paste in the link?

@stereotype441 @aam I have fixed the link now. It was included but I marked it incorrectly in the references section. Don't look to much at the prototype though - it is very hodgepodge. I am implemented just enough to get my samples running.

RE: @lrhn

since the loop is expanded and the elements of the list is inserted into the body as expressions, it is really looping over expressions, not values.

That's not what the proposal proposes. Please see above (emphasis added):

When @konst is applied to loop iteration variables it instructs the compiler to expand the loop at compile time by first computing the sequence of values for that iteration variable, then cloning the body for each value in order and substituting iteration variable with the corresponding constant.

So no expression business - you substitute variable with a constant value. This guarantees the following property: if @konst evaluation succeeds, then executing the same code using reflection will have the exact same behavior.

I do have an interest in AST based metaprogramming, but this proposal is explicitly not about it, because you can't avoid expanding AST templates.

Regarding the second comment about fields vs getters.

1. I don't see any reason to pretend that fields and getters are the same when accessing this information reflectively. They quack the same - but they are not the same in the text of the program. I don't agree that they should be seen in the same way at this level of introspection. Neither dart:mirrors nor analyzer model tries to pretend that they are the same. Yes, this might change behavior of the function when user changes field to getter or other way around. And yes, I think this is a correct capability to have - reflection gives you access to the program structure at a more precise level.
2. Regarding inherited fields: I think¹ you probably need explicit supertype chain walk. TypeInfo.fields only gives you fields of this exact type, not those of its superclass.
3. Re-privacy: we can choose to both restrict and allow access to private fields. e.g. I think it is okay to say that field.getFrom(o) errors if current library does not have access to the field.

RE: @TekExplorer

Wouldn't it be simpler to simply expand access to const,

I think you are missing one crucial piece: fromJson and toJson are not constant functions themselves and they can't be computed in compile time. Usually you give them objects that are only know in runtime. This means you can't just say toJson() const and call it a day. toJson which can only be called on a constant object is not very useful.

These functions are partially² constant - if some arguments (e.g. specifically T) are know in compile time then you can produce a version of toJson and fromJson specialized for that T.

That being said there is actually a way to make this model work, but it is not going to be pretty. You need to manually split constant and non-constant part of the toJson and fromJson computations. You can for example do something like:

typedef JsonMap = Map<String, Object?>;

/// Make serializer for T - this is constant part.
JsonMap Function(T) toJsonImpl<T>() const {
  final write = (JsonMap map, T o) {};
  for (final field in T.fields) {
    final previous = write;      
    write = (JsonMap map, T o) {
      previous(map, o);
      map[field.name] = field.getFrom(o);
    };
  }

  return (o) {
    final m = <String, Object?>{};
    writer(m, o);
    return m;
  };
}

class A {
  JsonMap toJson() => (const toJsonImpl<A>())(this);
}

But this has a bunch of draw-backs:

• It requires developer to expect compiler to perform a bunch of optimizations
  to arrive to the specialized code e.g. the chain of write callbacks needs to
  be fully inlined and compiler needs to fold field.name and field.getFrom
  accesses. If optimizations don't happen you end up with code which performs bad
  and potentially has bad code size (because full specialization is required to
  erase reflective metadata). That's why expectations around compile time expansion
  and specialization are encoded directly into semantics of @konst. If that's
  the case developer can reason about the code they get in the end (and they also
  get an error if compiler can't fully specialize the code).

• Significant mental gymnatics is required to arrive to this. Splitting the code
  into constant and non-constant part is inconvenient: consider toJson(this) vs
  (const toJsonImpl<A>())(this). To make matters worse correctly structuring
  your const part is non-trivial, e.g. you might be tempted to write

  JsonMap Function(T) toJsonImpl<T>() const {
    final fields = T.fields;
    return (o) => {
        for (var field in fields)
          field.name: field.getFrom(o),
      };
  }

  But this expects even more from the compiler: it now needs to unroll the
  loop to specialize the code and eliminate reflective metadata. That's why
  @konst has special provisions for loops for example.

RE: @schultek

I should rewrite the section about invoke a bit. It is not just for passing
type parameters around. You also need to be able to express invoking functions
with constructed argument lists. Consider: fromJson case which needs to
invoke constructor with named parameters.

I think invoke can be avoided only if we supported some form of spread
(...) into function arguments (including spread into type parameters).

Another reason to add invoke is that I would like to avoid any changes to
the Dart language itself. I think it makes this much simpler feature to ship.

Footnotes

1. Reflection APIs (and JSON example) does not try to be complete or precise - they serve to illustrate the approach. ↩

2. partially constant is not a real term, I just want to draw a parallel with partial evaluation ↩

[lrhn]

Ack, I did misunderstand how konst-loops work.

They are only over constant values (or at least "konstant values"), in which case duplicating the value is not an issue.

I don't see any reason to pretend that fields and getters are the same when accessing this information reflectively.

It's definitely possible to allow reflection only on instance variables, and not getters, but it means the code using reflection may change behavior, or even break, if someone changes a variable to a getter or vice versa, or adds implementation details like an int? _hashcodeCache;.

I'll absolutely insist that the breaking change policy says that reflection is not supported by any platform class unless it's explicitly stated as being supported. If someone complains that their code breaks in Dart 3.9.0 because I changed late final int foo = _compute(); to final int? _foo; int get foo => _foo ??= _compute();, they will not be allowed to block the change.
(Which likely also means lints and opt-in markers to ensure that no internal code depends on reflection of types that are not guaranteed to support it. Effectively it must be an opt-in feature. Might as well make it one from the start, so a class must be annotated as @reflectable to be allowed to be reflected.)

Also means that you can't reflect anything if all you have is an abstract interface type. It's all getters.
You can only reflect a concrete class type.

Accessing inherited fields explicitly through a TypeInfo.superClass makes sense. You'd want such a property too, then. Or rather, the declared superclass (the one in the extends clause), not the actual super-class. I expect that mixin applications counts as part of the class declaration, even if they are technically superclasses.
(Which is also not guaranteed to be stable over time, the superclass of an object can change without its API changing.)

Re-privacy: we can choose to both restrict and allow access to private fields. e.g. I think it is okay to say that field.getFrom(o) errors if current library does not have access to the field.

I don't believe "current library" is a viable or useful distinction.
It makes it impossible to rely on helper functions to do the actual invocation if those helper functions are in a different library.
The getFrom function is a konst function, so you can't tear it off, but you can pass the field object to another konst function and invoke it in there, perhaps a helper function in another library.

Map<String, Object?> getAllFields<@konst T>(@konst List<FieldInfo<T, Object?>> fields, T receiver) => 
    {for (@konst f in fields) f.name: f.getFrom(receiver)};

If there are some fields you can only access from "the same library", this helper function won't work.

If the implementation performs reflection at runtime, it would implement getFrom as a function which just accesses the actual getter on the argument object. That function can't see where it's called from.
(And don't say "look on the stack"! 😉)

I would at least make it possible to not get private members, and having to opt in to it. (But if a class needs to opt in to being reflectable to begin with, they could choose which kind of reflectable to allow, and maybe even for which individual fields. As long as the default @reflectable is a good default, like all non-private instance variables, you won't need to write more in most cases.)

So, the crux is that anything marked with @konst (when supported) is computed/expanded away at compile-time so that there is nothing @konst left. If a function has a parameter marked @konst (including this), it must be invoked at compile-time, and that argument (or all arguments?) is konst-evaluated too, and the body is effectively inlined at the call point. The function cannot be invoked at runtime, because it doesn't exist at runtime.
A konst for/in is over a konst list, and is unrolled.
It'll be a compile-time error if something is used in a way that would prevent konst-evaluation.
(And konst evaluation can't access any mutable global state.)

It's not a languge feature because the same code can be evaluated at runtime. It just requires some reflective functionality from the runtime system, but which can still only be invoked with values that could be known at compile-time.

[tatumizer]

Still don't understand why you need to use weird @K-words in

Map<String, Object?> getAllFields<@konst T>(@konst List<FieldInfo<T, Object?>> fields, T receiver) => 
    {for (@konst f in fields) f.name: f.getFrom(receiver)};

where you could write quite legibly

comptime Map<String, Object?> getAllFields<T>(List<FieldInfo<T, Object?>> fields, T receiver) => 
    {for (f in fields) f.name: f.getFrom(receiver)};

Also: in zig, the variable declared as const can be set in comptime, but it's still available in runtime. The meaning of @konst, on the other hand, is unclear: if you declare @konst x, will it be available in runtime?

The concept of comptime was formalized in zig after years of bikeshedding. If you try to borrow just some parts of it, you may eventually realize why you needed other parts, too 😄

[mraleph]

@lrhn

I'll absolutely insist that the breaking change policy says that reflection is not supported by any platform class unless it's explicitly stated as being supported.

I think this is fine. We can certainly change the definition of the breaking change to accommodate this. Note that changes which you describe do already break programs which use analyzer and mirrors - but we don't consider them breaking.

Also means that you can't reflect anything if all you have is an abstract interface type. It's all getters.

You can get methods (including getters), but not fields from such type. I think that's okay.

Which likely also means lints and opt-in markers to ensure that no internal code depends on reflection of types that are not guaranteed to support it.

Not everything is accessible through mirrors - e.g. we do restrict access to private members of dart:* libraries even though we otherwise allow access to private members. I think similar restrictions can be placed on compile time reflection.

So, the crux is that anything marked with @konst (when supported) is computed/expanded away at compile-time so that there is nothing @konst left. If a function has a parameter marked @konst (including this), it must be invoked at compile-time, and that argument (or all arguments?) is konst-evaluated too, and the body is effectively inlined at the call point. The function cannot be invoked at runtime, because it doesn't exist at runtime. A konst for/in is over a konst list, and is unrolled. It'll be a compile-time error if something is used in a way that would prevent konst-evaluation. (And konst evaluation can't access any mutable global state.)

It's not a languge feature because the same code can be evaluated at runtime. It just requires some reflective functionality from the runtime system, but which can still only be invoked with values that could be known at compile-time.

Yep, I think that's precisely a feature I propose. Except "all arguments?" part. Only @konst arguments are forced to become constants.

@tatumizer

Still don't understand why you need to use weird @K-words in

You should not overindex on @konst here - you need to see the essence of the feature. The syntax is secondary and it can change. We can introduce a special keyword in the same locations where @konst is written in the proposal. I've chosen @konst for two reasons:

• it is a valid syntax already, meaning that we don't actually need any language changes (like creating new keywords) to introduce this feature.
• we do have a tradition of using metadata (i.e. @pragma('...')) for features which are toolchain specific - which is part of my proposal as well.

[Wdestroier]

At first glance, this sounds like a really cool idea. I'm curious to play around with it more and try to understand its uses and limitations.

A few interesting examples imo:

• CRUD generator

  1. Write a model class, Book for example.
  2. Generate a repository class to save, find, update and delete the model.
  3. Generate a REST controller with the GET, POST, PUT and DELETE operations.
  4. Check if the /book/create and other endpoints are working.

• Type-safe ORM

  1. Write an entity class, Book for example.
  2. Support a fluent query syntax that mimics Iterable/List/Map operations (books.where((b) => b.reviews > 3).orderBy((b) => b.Url).toList();).
  3. Translate the operations to generated code with SQL.
  4. Check if basic and complex expressions work.

Example:

Future<List<LibrarySectionStatistics>> findUnresolvedIssuesGroupedBySectionOrderedByIssueCount() async {
  return await _libraryDbContext.bookIssues
      .asNoTracking()
      .where((bookIssue) =>
          bookIssue.status == BookIssueStatus.pendingOrInProgress &&
          bookIssue.resolutionDate == null &&
          bookIssue.section != null &&
          bookIssue.bookId != null)
      .groupBy((bookIssue) => bookIssue.section)
      .map((sectionAndBookIssues) => LibrarySectionStatistics(
            section: sectionAndBookIssues.key,
            totalIssueOccurrenceCount: sectionAndBookIssues
                .sum((bookIssue) => bookIssue.occurrenceCount),
            uniqueIssueOccurrenceCount: sectionAndBookIssues.length,
          ))
      .orderByDescending((result) => result.uniqueIssueOccurrenceCount)
      .toList();
}

would generate the translated SQL:

SELECT 
    Section AS Section,
    SUM(OccurrenceCount) AS TotalIssueOccurrenceCount,
    COUNT(*) AS UniqueIssueOccurrenceCount
FROM 
    BookIssues
WHERE 
    Status = 'PendingOrInProgress'
    AND ResolutionDate IS NULL
    AND Section IS NOT NULL
    AND BookId IS NOT NULL
GROUP BY 
    Section
ORDER BY 
    COUNT(*) DESC;

and generate the code to map the results back to objects.

[mraleph]

@Wdestroier Your Dart code is an order of magnitude less readable than SQL code. Which makes me doubt this sort of use case is something I would want to care about...

I think it is questionable API design if you want .groupBy((bookIssue) => bookIssue.section) to become GROUP BY Section. What about .groupBy((bookIssue) => sqrt(bookIssue.year) / bookIssue.numberOfPages + 1) should that become GROUP BY SQRT(Year)/NumPages+1? If query is going to be translated into SQL then it should be written in SQL (or built using SQL query builder). Writing it in Dart and then translating it to SQL is meh.

That being said. I think it is a valid question if we eventually want to support some form of expression trees or way to interact with AST from @konst computations. I don't think this needs to be included into the MVP of this feature though. Simple expression trees can already be extracted using special marker objects to construct an AST - and this can be used for DSL construction. But again maybe one should not do that.

Example of using custom marker objects to extract expression trees

sealed class ColumnExpression {
  ColumnExpression operator +(Object other) {
    final rhs = switch (other) {
      final int v => ConstantValue(v),
      final double v => ConstantValue(v),
      final ColumnExpression e => e,
      _ => throw ArgumentError('other should be num or ColumnExpression'),
    };
    return BinaryOperation('+', this, rhs);
  }

  ColumnExpression operator /(Object other) {
    final rhs = switch (other) {
      final int v => ConstantValue(v),
      final double v => ConstantValue(v),
      final String v => ConstantValue(v),
      final ColumnExpression e => e,
      _ => throw ArgumentError('other should be num, String or ColumnExpression'),
    };
    return BinaryOperation('/', this, rhs);
  }

  String toSql();
}

ColumnExpression sqrt(ColumnExpression expr) => UnaryOperation('SQRT', expr);

final class ConstantValue<T> extends ColumnExpression {
  final T value;
  ConstantValue(this.value);

  String toSql() => '$value';
}

final class ColumnReference extends ColumnExpression {
  final String ref;
  ColumnReference(this.ref);

  String toSql() => '$ref';
}

final class BinaryOperation extends ColumnExpression {
  final String op;
  final ColumnExpression lhs;
  final ColumnExpression rhs;
  BinaryOperation(this.op, this.lhs, this.rhs);

  String toSql() => '(${lhs.toSql()} $op ${rhs.toSql()})';
}


final class UnaryOperation extends ColumnExpression {
  final String op;
  final ColumnExpression lhs;
  UnaryOperation(this.op, this.lhs);

  String toSql() => '$op(${lhs.toSql()})';
}

class BookIssuesColumns {
  const BookIssuesColumns();

  ColumnExpression get year => ColumnReference('Year');
  ColumnExpression get numberOfPages => ColumnReference('NumberOfPages');
}

void main() {
  print(((bookIssue) => sqrt(bookIssue.year)/bookIssue.numberOfPages + 1)(const BookIssuesColumns()).toSql());
  // SQRT(Year)/NumberOfPages + 1
}

[Wdestroier]

I prefer Dart's syntax (operations are not out of order), but I have other arguments, for example: with an ORM the database can be changed from Postgres to MongoDB without rewriting all the SQL code, because changing the provider would be enough. Perhaps this code is more readable:

Abbreviated code

Future<List<LibrarySectionStatistics>> findUnresolvedIssuesGroupedBySectionOrderedByIssueCount() async {
  return await _libraryDbContext.bookIssues.asNoTracking()
      .where((b) =>
          b.status == BookIssueStatus.pendingOrInProgress &&
          b.resolutionDate == null &&
          b.section != null &&
          b.bookId != null)
      .groupBy((b) => b.section)
      .map((g) => LibrarySectionStatistics(
            section: g.key,
            totalIssueOccurrenceCount: g.sum((b) => b.occurrenceCount),
            uniqueIssueOccurrenceCount: sectionAndBookIssues.length,
          ))
      .orderByDescending((r) => r.uniqueIssueOccurrenceCount)
      .toList();
}

What about .groupBy((bookIssue) => sqrt(bookIssue.year) / bookIssue.numberOfPages + 1)

Apparently sqrt isn't supported by Entity Framework, but pow(x, 0.5) is supported.
From this StackOverflow answer: var place = db.Places.FirstOrDefault(x => Math.Pow(x.Lat, 0.5) > 0);.

I feel like writing ColumnExpression sqrt(ColumnExpression expr) => UnaryOperation('SQRT', expr); ends with the magic vibes. Thank you for the example.

I don't think this needs to be included into the MVP of this feature though.

True, I agree. I gave a very difficult example I could think of 😄. Creating something equivalent to EntityFramework would probably require effort from the Dart team, it is very hard to implement without special language or compiler features imo.

[mraleph]

@Dangling-Feet you don't seem to be providing feedback for this particular proposal and seem to instead propose macro system similar to one which was already explored and shelved for a variety of reasons. Such generic comment is better posted to #1482

[Dangling-Feet]

@mraleph Thank you.

[crefter]

Waiting good news ;) (We need something like this which helps create toJson/fromJson, (ORM?), data classes, configs and etc without wasting time for generation) GL!

[lucavenir]

I might be way too ignorant to comment this issue, but reading this scares me:

In this model, developer might encounter compile time errors when building release application which they did not observe while developing - as development and deployment toolchains implement different semantics.

This is really scary.
Mainly because - as far as I understand - @konst wouldn't be *actually testable: tests run in development mode, right?
Oh boy I can't wait to have a fine running app that works on my computer™, only to see it break in production with incomprehensible errors on my crashlytics console 😜

I think that's an acceptable price to pay for the convenience&power of this feature.

I don't think so. Again I'm not smart enough to have a strong opinion on this, but I feel like this is a deal breaker for me. I think of my clients and I just can't afford the consequences of this.

I don't know, maybe I'm misreading these two sentences. But if I read these correctly, wouldn't it be possible to avoid this problem, somehow? Can't the trade-off be put somewhere else? 🥺

[cedvdb]

1. In regards to serialization, the implementation above would fail in the presence of a private field. json[field.name] won't work for this simple class if the json is {bar: 3}.

class Foo {
  final int _bar;
  this({ required int bar}) : _bar = bar;
}

So you'd have to check whether the name starts with _.

2. How do I generate code like an enum or a copyWith ?

Oh boy I can't wait to have a fine running app that works on my computer™, only to see it break in production with incomprehensible errors on my crashlytics console

@lucavenir It wouldn't go into production, because it wouldn't compile.

[tatumizer]

In regards to serialization,..

In the same regards, serialization is such a rabbit hole - you can get stuck there for life. Here's the list of annotations to control serialization in a popular jackson framework. This list is incomplete (stuff gets added all the time), and it cannot be made complete in principle because of its infinite size.

[lucavenir]

@lucavenir It wouldn't go into production, because it wouldn't compile.

Oh damn, you're right. Woops!

Still, my cortisol levels aren't lowering; the "works on my machine" issue becomes a build time issue, which can potentially back-propagate to my codebase. And this can potentially black-swan my software production's lifecycle.

Did I get this right? Let me imagine a scenario.

For example I could potentially write my application, test it, use it in debug mode (JIT compiler), be happy with it.
Then, I'd have to ship it, so I'd build it with the AOT compiler, and... poof everything breaks.
It turns out I misused some @konst annotations; touching that can potentially make me re-think my codebase.
Essentially, I've set-up traps in my codebase that I must be really careful with.
I'd start fearing my own code, which we know is the worst for a codebase (my inner software engineer is screaming at this thought).
I'd also have "no way out" - no way of "future-proofing" the fixes against such issues.

I fear this is still a deal-breaker for me. I'm not convinced this is a "small price to pay", yet.

[benthillerkus]

For example I could potentially write my application, test it, use it in debug mode (JIT compiler), be happy with it.
Then, I'd have to ship it, so I'd build it with the AOT compiler, and... poof everything breaks.

How's that different from today?

It turns out I misused some @konst annotations; touching that can potentially make me re-think my codebase.
Essentially, I've set-up traps in my codebase that I must be really careful with.

I'd also have "no way out" - no way of "future-proofing" the fixes against such issues.

You control the buttons you press

[lucavenir]

How's that different from today?

It's different because as of today the average developer won't shoot its foot like that; AFAIK the only way to set-up a footgun like this, today, is to use @pragma annotations, but realistically no one uses that annotation as of today.
Keep in mind that it's discouraged to carelessly play with @pragma annotations unless you know what you're doing - that stuff is just not meant to be used from "us" final users (devs).

There are "general purpose" VM-related annotations, but again no one really uses them in their day-by-day code, isn't it? Also these apply to the VM and not to the AOC compiler AFAIK.

The point is - this proposal kind-of suggests that @konst is meant to be used to achieve basic functionalities (e.g. toJson), opening this door to everyone.

You control the buttons you press

Sure. Until you don't.
Imagine you're pressing buttons, everything turns out to be good while developing and testing, green lights everywhere, and then we build... woops every assumption you've made is wrong!
Do we actually control that?
(please correct me if this scenario isn't possible)

[lrhn]

fold the cost of reflection away by enforcing const-ness requirements implied by @konst

I think all compilers should enforce the const-ness requirements of @konst parameters, and should only differ in implementation strategy. That only konst-able expressions are arguments to konst parameters, and that konst-values are not arguments to non-konst parameters - because then we won't compile away the konst-ness.

That probably means that the @konst-requirements are checked by a common front-end, then the implementation is left to the backends, either just running normally and doing runtime reflection, or inlining the konst-invocations and specializing the operations on @konst parameter values, until there is no konst-marked code left in the program.

What is important is that a @konst value can be evaluated without (state-based) side-effects at compile-time, so that if it's evaluated at runtime, that is indistinguishable from being pre-computed at compile-time.
(May want to say something about whether konst-but-not-const values can be canonicalized. They probably shouldn't be.)

I guess it is still possible to get a runtime-error due to the the reflected data, things that can't be checked by static analysis alone, if you make assumptions like .members.single. That is only a compile-time error if evaluating the konst at compile-time. It will be a runtime error in development, it won't be silently accepted.

Basically, the gurantee is that if

• If it compiles with konst-as-compile-time, then it behave the exact same way with konst-as-runtime.
• If all konst-marked code runs without error with konst-as-runtime, then it will compile with konst-as-compile-time, and have the same behavior.

Then there should also be no issue with the code compiling and running successfully in development, but failing to compile with a production compiler. The development compilers would have failed to compile or to runt the code too, if the program contains anything that the production compiler would fail at.
(But if you have konst code that you never run or test, then it may fail to compile with a production compiler.)

[jakemac53]

Note that today it is possible (for many of our compilers and the analyzer) to compile/analyze a library given only the API level information of its dependencies (ie: the analyzer summary or kernel outline).

If any library can invoke any dependencies code at compile time that is no longer the case, and this has significant implications for invalidation, especially in scenarios such as blaze. This is exactly why we previously tabled the enhanced const feature.

We could alleviate these concerns potentially by separating out the imports into const and non-const imports (via some syntax), and in the blaze world we would probably want this to correspond to const and non-const deps in the blaze rules.

This would allow the build systems to know which dependencies need to be included as full dill files (and for the analyzer, probably just as Dart source files), instead of summaries, so that the cost can be paid only for the dependencies that are actually used at compile time. This wouldn't be a perfect solution though because all transitive dependencies of those dependencies would also have to be included as full dills, since we don't know what will actually be used.

[mraleph]

@jakemac53 I think I should include some explicit remarks about compiling/analyzing against outlines in the proposal. That's one of the reasons I formulated proposal in the way I did, but I did poor job stating it. Only AOT compilation toolchains will need to do @konst evaluation because they have access to the closed world anyway.

[jakemac53]

Is the idea then that the analyzer would never execute these functions at all? That would certainly speed up analysis, with the trade-off that you may have compile time errors that the analyzer doesn't report.

It could only execute these for open files or something like that though, possibly.

In general I think the idea of not actually executing these at compile time during the development cycle is an interesting idea, although we definitely will have to be thoughtful about how hot reload will work, and also bear in mind that it will slow down the startup of all dart apps potentially (in dev mode) - you have to evaluate all these every time an app launches if I understand correctly, instead of having the evaluation be performed once at compile time and then cached (hopefully) for rebuilds.

[lrhn]

There is no extra execution on start-up. If the @konst code is not executed at compile-time, it's executed at runtime as completely normal code when control gets to it. The Analyzer can analyze the code using the same rules as any other code - plus enforcing whatever extra restrictions there are on @konst-annotated code, but that should be local to each invocation of a @konst function.
The analyzer doesn't need to execute anything, it just checks that the types are right and that @konst parameters get arguments that are konst-evaluable.

It's normal code that can be constant evaluated/specialized to improve tree-shaking.

(Well, normal code that can do a limited kind of reflection.)

[jakemac53]

There is no extra execution on start-up. If the @konst code is not executed at compile-time, it's executed at runtime as completely normal code when control gets to it.

It still happens at runtime either way, even if it is only on first access. Maybe it is fine, I don't really know. It depends on how prevalent the usages are. Probably roughly equivalent to using mirrors today.

[safasofuoglu]

Love the proposal. (I always enjoy and appreciate long-form writing from @mraleph).

I am currently working on combining a native game engine with the usability and ergonomics of Dart.

This proposal looks invaluable from that standpoint.

[munificent]

This is a really cool proposal.

@jakemac53 and I did do some investigation of C++ and Zig early on in the macro design process. That's partially where "enhanced constant functions" came from. As you note, it has a lot of really nice properties. A couple of thoughts:

Fields versus getters

I agree with @lrhn that allowing reflective access to the fields of an arbitrary class breaks encapsulation and uniform access. Personally, I think uniform access is an important feature of Dart. I like that class authors are free to change fields to getters and vice versa without breaking clients. More importantly, I like that class authors aren't encouraged to preemptively wrap every field in a getter just in case they need that abstraction later.

I believe we could address that while doing this proposal by restricting what you're allowed to introspect on. The simplest approach would be to say you can only introspect on the fields of the surrounding class. A little more powerful would be allowing something like "template mixins" where a mixin is allowed to introspect over the class its applied to.

In both cases, the class author has full control over whether the class is introspectable.

I wouldn't want to allow unbounded introspection over arbitrary types. Encapsulation is important.

No declaration-level metaprogramming

This feature works when the place where you need to do some metaprogramming is only inside a function body. For example, it works well for methods like toJson() and fromJson() where the signatures are fixed and it's only the body that needs to be reflective.

But it doesn't handle other use cases like copyWith() where you want to use metaprogramming to procedurally build the parameter list itself. It doesn't handle some "value type" use cases where you want to procedurally define a list of fields.

It may be that we can simply live without supporting those use cases. Or it may be that some kind of "record spread" support in parameter lists and/or field declarations could help.

But I suspect that this is a severe enough limitation that it means the feature wouldn't carry its weight. It's still quite a lot of complexity and in return you only get the ability to do metaprogramming inside function bodies.

If you look around at the places where users are using code generation in Dart today, it's very often adding new declarations, and this wouldn't help for that.

[tatumizer]

In zig, you can generate the definition of a new struct (=class) in comptime. See the examples here: https://mht.wtf/post/comptime-struct/
The idea is that you write a function that may take a type descriptor as a parameter, and return another type descriptor that gets interpreted as a declaration of a new struct, in a pure functional manner.
Not sure if this approach is applicable to dart, but who knows.

[benthillerkus]

I'm curious if you could use this to create a version of flutter_hooks that passes each Hook a hidden random key that is statically generated via @konst.

That way the hooks implementation could use the keys to detect divergence, but also to converge again later; ie you could use hooks inside of if statements, similar to Jetpack Compose.
You still wouldn't be able to use them inside of loops, because you'd then call the same hook with the same static key multiple times (at least if you're not using @konst loops), but as far as I can tell you cannot do that in Jetpack Compose either.

Now to be honest I'm not exactly sure how it would work with the proposed system, but I think each useXY method would have to be annotated to run statically. Inside of them you could then maybe just use Math.random() or Object() to get a key and then return the Hook object with the key passed into the constructor.

Since the key is generated during @konst it would be specific to each useXY callsite, so that the same useXY callsite would always return a Hook with the same key on every build().

Now, would that work? Or something like this?
cc @rrousselGit

[benthillerkus]

Oh another issue could be that since numbers are slightly differently implemented in dart2js than the 'native' Dart targets, that the result of a static calculation or bit shift operation could be different than the same operation in JavaScript at runtime.

[lrhn]

Number semantics are already an issue in const evaluation. The compilers do constant evaluation that matches their runtime behavior.
So far there is no compiler which targets both JS and native.