Improvements targeting serialization & data

[davidmorgan]

We'd like to make some combination of language and tooling improvements to better support serialization & data modeling.

More details to follow. We'll be asking for input on what people want to see--those interested can watch this issue for updates.

[bivens-dev]

I have two main ones that I’m particularly interested in.

This proposal from @munificent a few years ago: https://github.com/dart-lang/language/blob/main/working/tagged-strings/feature-specification.md

And I would really like to see proper core language support for the RFC 8949 Concise Binary Object Representation specification including code generation for its sister specification Concise Data Definition Language (CDDL)

[bivens-dev]

Oh and one last one that I would really love to see although I don’t know it fits the idea of a language or even tooling change more just something that I think is missing from the ecosystem which is a good RDF (Resource Description Framework) library.

After many years of work the newest version is due to be finalized quite soon I believe.

It’s a hugely popular and powerful way to model data in an interoperable and language neutral way and would pair incredibly well with a lot of the AI work Google is bringing to Flutter and elsewhere.

[davidmorgan]

Thanks @bivens-dev ... what would core language support for RFC8949 look like? I mean, compared to just doing it with codegen.

Technical suggestions for specific things we could do/support are useful, thanks; it'd also be good to hear about specific use cases that don't work well today that would benefit.

[bivens-dev]

Thanks for the clarifying question. Reading it back now I agree the phrase "core language support" is incredibly ambiguous, my apologies for that.

Specifically what I mean in the case of RFC8949 that there would exist a spec compliant streaming encoder and decoder that lived within dart:convert that allowed for converting between native Dart data types and CBOR. Note there is something like this with limited community support but it's a one person project and I am probably making the argument here that in terms of the strategic importance[1] of this as a universal serialization format that probably represents an unacceptable risk of sorts to many organizations.

That would be independent of the Concise Data Definition Language (CDDL) point I mentioned which I guess is the CBOR equivalent of a protobuf file / language and is what would be used as the basis of the code generation.

I would probably also make a similar argument for RDF in that I think it is already seeing incredibly strong adoption not just among businesses but also amongst governments. Some examples of that might include the Australian Government Linked Data Working Group and European Union's Linked Open Data.

Right now, I'm not really aware of any good solutions for working with RDF data at all within the Dart community and that too feels like something that will become a real problem before too long. Once again, I would make the same argument that given the level of trust people and organizations are going to need around a topic like that, community support doesn't feel like an appropriate option.

I think in concrete terms it too would ultimately look like a library that in addition to providing Dart native representations of key concepts like graphs, tuples, resources etc would also include a Codec to convert to and from the most common serialization formats.

Hopefully, that helps clarify things a bit more and generally provide some additional context as to why I think they are important both in terms of capabilities but also why I think they need certain quality guarantees that I don't think can actually come from the community at this time.

[1] My understanding of this is that CDDL is now the new defacto standard for representing data types and has been for some time already for all IETF standards. Some relevant examples using CBOR that immediately come to mind both from IETF and others include:

1. RFC 8152: CBOR Object Signing and Encryption (COSE)
2. RFC 8392: CBOR Web Token (CWT)
3. RFC 8742: Concise Binary Object Representation (CBOR) Sequences
4. RFC 8746: Concise Binary Object Representation (CBOR) Tags for Typed Arrays
5. RFC 8943: Concise Binary Object Representation (CBOR) Tags for Date
6. RFC 7252: The Constrained Application Protocol (CoAP)
7. RFC 9200: Authentication and Authorization for Constrained Environments
8. RFC 8428: Sensor Measurement Lists (SenML)
9. W3C: Recommendation: WebAuthn
10. Oasis: Open Command and Control (OpenC2)
11. W3C: Web of Things (WOT)
12. W3C: Digital Credentials API Which is also coming under various regulatory frameworks such as European Union's eIDAS 2.0.

[mosuem]

In case it wasn't on the radar yet: https://github.com/schultek/codable
cc @schultek

[davidmorgan]

Re: "tooling improvements", I plan on working on build_runner performance: dart-lang/build#3800

[Mike278]

ORs need the record treatment!

What?

Well one of the problems addressed by records is that if you decide "actually this function should return this type AND this other type" then you had to make a new class, which was a pretty steep syntatic cliff.

Currently, if you decide "actually this function should return this type OR this other type" then you make a sealed hierarchy of classes, which is a similarly steep cliff.

I don't think this needs a super powerful solution like (untagged) union types. IMO the desirable property of union types for most people is not that they save you from having to pick a name for each variant, but that the syntax is concise and inline.

One of the things I was planning on using macros for was seeing how concise I could make the syntax for defining what would eventually generate a sealed hierarchy of classes. I think the most relevant issues are #3021 and #2364 but those both still require writing new top-level definitions - the ideal would be writing it inline. For example:

// strawman syntax
.ConnectResult(.AuthExpired | .Offline | .Success({bool stable, Connection connection})) connect() {

}

// would internally generate
sealed class ConnectResult {}
class AuthExpired implements ConnectResult {}
class Offline implements ConnectResult {}
class Success implements ConnectResult {
  final bool stable;
  final Connection connection;
  Success({required this.stable, required this.connection});
}

[lukepighetti]

one of the key benefits of sum types is you can evolve an api without breaking changes

today: build(String title)

future: build(String | Widget title)

[tatumizer]

// strawman syntax
.ConnectResult(.AuthExpired | .Offline | .Success({bool stable, Connection connection})) connect() {

}

This is functionally equivalent to rust's enums.

Sum types like (A | B) have their uses, too - one doesn't exclude the other.

[lrhn]

can evolve an api without breaking changes

Widening the parameters of an instance method is breaking if there is any subclass implementing the narrower parameter type.
Don't need general union types to break things, just going from void foo(String input) to void foo(String? input) will break every subclass which implements the interface, or which extends the class and overrides the member.

Changing an instance member signature to a subtype is breaking for subclasses.
Changing an instance member signture to a supertype is breaking for clients.
Only way to win is not to play. (Or have overloading.)

[rrousselGit]

^ That's arguable.
Most of the time, I'd use 0.x.0 bump when loosening the type of a parameter.

If we think about it for long enough, almost everything is breaking in some capacity.

For example, technically speaking modifying any function prototype is breaking ; even if they don't belong to a class/interface, because of type-inference.

Say a package makes such change:

-void function(String input) {};
+void function(String? input) {};

This would break apps that do:

final callbacks = [function];
callback.add((value) => print(value[0]));

But realistically, nobody does that.
Making a major version for such change would be bothering 99.9% of the users because maybe one person in world is impacted.

The same applies to class methods IMO.
Unless the method is designed to be extended, I wouldn't consider loosening a parameter type breaking.

[lrhn]

Unless the method is designed to be extended, I wouldn't consider loosening a parameter type breaking.

I might be colored by working on the platform libraries, where we can usually assume that anything a user can do, someone does.

There are classes that were never intended as interfaces, and methods that were never intended to be overridden in an extending subclass. Widening the parameter type of such a function on such a class can probably be ruled to be "non-breaking in practice". That's what the Dart breaking change process is for, deciding whether a breaking change is expected to be actually breaking on a significant scale.

(Doesn't help us if it actually breaks something anyway.)

But, getting back on track, union types won't help if a change of actually breaking. They're just a wider parameter type.

[Mike278]

can evolve an api without breaking changes [...]
future: build(String | Widget title)

Don't need general union types to break things

And you also don't need general union types to evolve an api without breaking changes - i.e build(Title someSealedClass) - but it's quite a leap in power (and cost in syntax) compared to a single concrete type.

Records were great because they added an intermediate step in power. Perfect for when you don't need all the capabilities of a full blown class (constructors, inheritance, methods, mutable state, etc) - you need to AND a few type:label pairs together. Don't pay for what you don't need.

It would be nice if there was a similar intermediate step before "write out a sealed class hierarchy" for when you just need to OR a few type:label pairs together.

General union types are definitely a solution, but IMO they're unnecessarily powerful for this specific problem to justify their cost.

That's why I was thinking of something that has the nice concise inline "structural" syntax of union types, but would desugar to a sealed class hierarchy. Similar to how records are more or less syntax sugar for simple classes (runtime considerations notwithstanding).

[santiagos01]

While having sum type / Rust-like enum would certainly be great, honestly I find it more painful at the moment to have to resort to codegen (e.g., with dart_mappable or freezed) just to have proper basic functions like JSON serde and copyWith (which are significantly used by typical apps communicating with API server). I really wish working with data types can be as simple as in the JS/TS world...

Besides, I think it would be nice to have things like these built natively (or included in the standard library), like the path that Go took with struct tag, encoding/json package, and its copy-by-value semantic (of course without the need for any codegen).