Skip to content

[native_assets_cli] Unify Metadata with Assets #2164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 4, 2025
Merged

Conversation

dcharkes
Copy link
Collaborator

@dcharkes dcharkes commented Apr 3, 2025

Bug: #1251

This PR adds support for two things:

  1. Routing assets from build hook to build hooks.
  2. Adding a MetadataAsset for sending metadata between hooks.

This enables two use cases:

  1. Deprecating the current bespoke metadata system in favor of one built on top of the new routing with the new asset type.
    Example: pkgs/native_assets_builder/test_data/package_with_metadata/hook/build.dart + pkgs/native_assets_builder/test_data/package_reading_metadata/hook/build.dart
  2. Exporting a dynamic library from one package, and using the native linker to link against it in dependent library.
    Example: pkgs/native_assets_builder/test_data/reusable_dynamic_library/hook/build.dart + pkgs/native_assets_builder/test_data/reuse_dynamic_library/hook/build.dart

To accommodate this change the String? linkInPackage is replaced with Routing routing everywhere. This has three subtypes:

  1. BundleInApp(), equivalent to linkInPackage: null,
  2. Tolinker(String packageName), equivalent to linkInPackage: packageName, and
  3. ToBuild() which makes the asset available in the BuildInput of the direct dependencies.

Note that assets routed to other build hooks via ToBuild() also need to be routed to BundleInApp(). If another package uses the asset at runtime, it cannot re-bundle it itself because that would lead to double bundling if two packages use such asset.

(This leads to a weird quirk where we now have List<Routing> as a parameter for CBuilder because we want the asset to be added in two places. We should probably redesign CBuilder.run to not use the HookInput/HookOutput directly.)

Copy link

github-actions bot commented Apr 3, 2025

PR Health

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbols
License Headers ✔️
// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/jni/lib/src/third_party/generated_bindings.dart
pkgs/objective_c/lib/src/ns_input_stream.dart

@dcharkes dcharkes force-pushed the metadata-refactor branch from e2e2ef3 to 2f4d1c1 Compare April 3, 2025 11:07
@github-actions github-actions bot added the type-infra A repository infrastructure change or enhancement label Apr 3, 2025
@dcharkes dcharkes marked this pull request as ready for review April 3, 2025 11:38
@dcharkes dcharkes force-pushed the metadata-refactor branch from d6a45fa to 5514561 Compare April 3, 2025 12:06
@mkustermann
Copy link
Member

We reserve the right to invoke build hooks & link hooks once or multiple times. If we invoke it multiple times we may add different values for buildAssetTypes.

@dcharkes Is the new MetadataAsset a required asset type that any build hook is allowed to use? (i.e. protocol mandates it to be in buildAssetTypes)?

@dcharkes
Copy link
Collaborator Author

dcharkes commented Apr 3, 2025

We reserve the right to invoke build hooks & link hooks once or multiple times. If we invoke it multiple times we may add different values for buildAssetTypes.

@dcharkes Is the new MetadataAsset a required asset type that any build hook is allowed to use? (i.e. protocol mandates it to be in buildAssetTypes)?

You can output any asset type in ToLinker and ToBuild. The checking for asset types is only for the BundleInApp Routing. So, it does not have to be in buildAssetTypes. Any unknown asset type can flow between two build hooks or between a build hook an a link hook.

@dcharkes dcharkes force-pushed the metadata-refactor-syntax branch from 5f12140 to b5729b9 Compare April 4, 2025 07:50
@dcharkes dcharkes force-pushed the metadata-refactor branch from 5514561 to 6041be6 Compare April 4, 2025 07:51
@dcharkes dcharkes force-pushed the metadata-refactor-syntax branch from b5729b9 to 3da9128 Compare April 4, 2025 08:04
@dcharkes dcharkes force-pushed the metadata-refactor branch 2 times, most recently from fff25a7 to 3525c01 Compare April 4, 2025 08:18
@dcharkes dcharkes force-pushed the metadata-refactor-syntax branch from 3da9128 to 0e1f6e1 Compare April 4, 2025 08:35
@dcharkes dcharkes force-pushed the metadata-refactor branch from 3525c01 to 28009f3 Compare April 4, 2025 08:35
@@ -32,7 +32,10 @@ void main(List<String> arguments) async {
output.assets.code.add(
tempBuildOutput.assets.code.single,
// Send dylib to linking if linking is enabled.
linkInPackage: input.config.linkingEnabled ? packageName : null,
routing:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an idea: We could avoid the input.config.linkingEnabled bool calls by instead having something like an input.config.routing, which can be a LinkingEnabledRouter which has a toLinker method, or a LinkingNotEnabledRouter which only has bundleInApp or toBuilder`.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like having a input.config.routing. 👍 That makes the API symmetric again.

I'm not sure that having to take the value from the input would make a very nice API. Especially for ToBuild or BundleInApp where you don't need to check the config. Let me give it a try.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a spin, but make the call sites even more verbose.

      routing: switch (input.config.routing) {
        final LinkingEnabledRouter r => r.toLinkHook(packageName),
        LinkingDisabledRouter() => const ToAppBundle(),
      },

Then the following is more readable:

      routing: switch (input.config.routing) {
        LinkingEnabledRouter() => ToLinkHook(packageName),
        LinkingDisabledRouter() => const ToAppBundle(),
      },

Current solution (this doesn't show the routing)

      routing: switch (input.config.linkingEnabled) {
        true => ToLinkHook(packageName),
        false => const ToAppBundle(),
      },

With routing added:

      routing: switch (input.config.routing.linkHooksEnabled) {
        true => ToLinkHook(packageName),
        false => const ToAppBundle(),
      },

True false is not very pretty. Let's try a when

      routing: switch (input.config.routing) {
        final routing when routing.linkHooksEnabled => ToLinkHook(packageName),
        _ => const ToAppBundle(),
      },

Also not that great.

Then the following is more readable:

      routing: switch (input.config.routing) {
        LinkHooksEnabled() => ToLinkHook(packageName),
        LinkHooksDisabled() => const ToAppBundle(),
      },

But it only works if that's the only option we ever want to add to the routing. It's only the Dart API, so we could do it, it's easy to refactor.

(P.S. If we want to go the route with the method, we should probably do the same with LinkModePreference and LinkMode.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try this in a more holistic way for everywhere where the input-output constraint could be made more explicit.

#2175

@dcharkes dcharkes force-pushed the metadata-refactor-syntax branch from 0e1f6e1 to cf34082 Compare April 4, 2025 13:10
auto-submit bot pushed a commit that referenced this pull request Apr 4, 2025
#2163)

Bug: #1251

This PR contains the JSON spec for:

* #2164

For more details see that PR.

This PR adds the syntax for passing assets from build hooks to build hooks.

build output `assets_for_build`:

```json
{
  "assets": [],
  "assets_for_build": [
    {
      "encoding": {
        "a_key": "some_value"
      },
      "some_key": "some_value",
      "type": "some_asset_type"
    }
  ],
  "assets_for_linking": {},
}
```

build input `assets` per package name:

```json
{
  "assets": {
    "some_package": [
      {
        "some_key": "some_value",
        "type": "some_asset_type"
      }
    ]
  },
  "config": {},
}
```

Secondly, this PR adds a syntax for metadata assets (assets that cannot be bundled but just ferry information between different hooks).

```json
      {
        "encoding": {
          "key": "foo",
          "value": "bar"
        },
        "type": "hooks/metadata"
      },
```

The schemas now check these.

And this is covered by tests.

Changes to the generated schemas:

* Override the new build input `asset` and build output `assets_for_build` with the `CodeAsset` and `DataAsset` in the generated extension schemas.

Notable changes to the syntax generator:

* Enable generating tagged enum classes if there are two fields (a tag, and some `encoding`) in the super class.
* Support `Object` fields (used for `MetadataAsset.value`).
Base automatically changed from metadata-refactor-syntax to main April 4, 2025 13:22
@dcharkes dcharkes force-pushed the metadata-refactor branch from 803df09 to 22b0a86 Compare April 4, 2025 13:47
@coveralls
Copy link

Coverage Status

coverage: 84.923% (+0.1%) from 84.822%
when pulling 2f240a9 on metadata-refactor
into 481af77 on main.

@dcharkes dcharkes merged commit 8a1f836 into main Apr 4, 2025
33 checks passed
@dcharkes dcharkes deleted the metadata-refactor branch April 4, 2025 14:24
github-merge-queue bot pushed a commit to flutter/flutter that referenced this pull request Apr 10, 2025
Updating the dart-lang/native dependencies to the ones published
yesterday.

Incorporates a fix for:

* dart-lang/native#2149
* Note that a bunch of tests were relying on ignoring build assets. The
tests have been fixed.

And the new `AssetRouting` syntax in the build hook for sending assets
to the link hook:

* dart-lang/native#2164

Does not include support for user-defines in the pubspec yet. Will do
this in a follow up PR.

* dart-lang/native#2165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants