Skip to content

Commit d8a84ea

Browse files
authored
Allow custom asset types in the build system (#1623)
This is the first PR that will increase the flexibility of the Dart Build system as well as makes it more layered. The main points being addressed in this PR are: * A bundling tool has to specify which asset types it supports. So we make `supportedAssetTypes` required everywhere. => Across various layers in the code base we make this required => We remove a baked-in `[CodeAsset.type]` fallback in various places (Making this explicit increases size of LOCs in this CL due to the many tests) * The core building infrastructure in `pkg/native_assets_builder/lib/*` no longer knows anything about `CodeAsset` or `DataAsset`. Instead it only knows about `EncodedAsset` which represents the type of an asset and it's json encoding. => The core still verifies certain protocol consistency (e.g. that a hook only emitted asset types that the config allows) => The higher levels pass the `supportedAssetTypes`. => The higher levels now also pass `buildValidator` / `linkValidator` that validate per-asset related things (e.g. the id of a data asset to start with the package name) for a given package. => The higher levels now also pass a `applicationAssetValidator` that validates consistency of assets across all packages (e.g. uniqueness of dylib filenames across all dependencies) => This centralizes the logic about per-asset-type information in the bundling tool (i.e. user of `package:native_assets_builder`). => Bundling tools now have to expand `CodeAsset`s in dry-run to all architectures. * The bundling tool (e.g. `flutter build`) doesn't have to implement validation logic itself, instead it will support certain asset types by importing the "bundling logic" for that asset types (which includes - for now - validation logic). * All the validation code now simply returns `ValidationError` which is a list of errors. If the list is empty then there were no errors. => This removes redundancy between a `success` boolean and the list of errors (it was successsfull iff there were no errors) => This simplifies code that aggregates errors from different sources. * Moves the `fromJson` / `toJson` towards be purely read/write a json schema and are not responsible for anything else. => The validation routines are responsible for validating semantics. * The hook writer API now changes to work on per-asset-type specific extension methods. For example what used to be ``` output.addAsset(DataAsset(...)); ``` is now ``` output.dataAssets.add(DataAsset(...)); ``` The `BuildOutput.encodedAsset` getter is (with this PR) temporarily visible to hook writers, which could use it like this: ``` output.encodedAssets.add(DataAsset(...).encode()); ``` but a following refactoring that increases the flexibility of the protocol more will also get rid of the user-visible `encodedAsset` getter (if we choose so). So this is only temporarily exposed. * `hook/link.dart` scripts have to mark any input files it's output depends on in the `LinkOutput.addDependency`. If the set of inputs is the same but one input's file changed then a `hook/link.dart` - doesn't have to rerun if it just drops all assets or outputs them unchanged - has to re-run if it copies a input, changes it's contents, merge with other inputs, etc => A `hook/link.dart` should - just like a `hook/build.dart` add dependencies if the inputs are used to produce the output => Before this was somewhat hard-coded into the `package:native_assets_builder`.
1 parent 0bf27dc commit d8a84ea

File tree

116 files changed

+1473
-1035
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

116 files changed

+1473
-1035
lines changed

pkgs/native_assets_builder/CHANGELOG.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,10 @@
1010
dart-lang/native repository to make it clear those are not intended to be used
1111
by end-users.
1212
- Remove link-dry-run concept as it's unused by Flutter Tools & Dart SDK
13-
- Bump `native_assets_cli` to `0.9.0`
13+
- Bump `native_assets_cli` to `0.9.0`.
14+
- **Breaking change**: Remove asset-type specific logic from `package:native_assets_builder`.
15+
Bundling tools have to now supply `supportedAssetTypes` and corresponding
16+
validation routines.
1417

1518
## 0.8.3
1619

pkgs/native_assets_builder/lib/src/build_runner/build_runner.dart

+72-74
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,25 @@ import 'build_planner.dart';
2323

2424
typedef DependencyMetadata = Map<String, Metadata>;
2525

26+
typedef _HookValidator = Future<ValidationErrors> Function(
27+
HookConfig config, HookOutputImpl output);
28+
29+
// A callback that validates the output of a `hook/link.dart` invocation is
30+
// valid (it may valid asset-type specific information).
31+
typedef BuildValidator = Future<ValidationErrors> Function(
32+
BuildConfig config, BuildOutput outup);
33+
34+
// A callback that validates the output of a `hook/link.dart` invocation is
35+
// valid (it may valid asset-type specific information).
36+
typedef LinkValidator = Future<ValidationErrors> Function(
37+
LinkConfig config, LinkOutput output);
38+
39+
// A callback that validates assets emitted across all packages are valid / can
40+
// be used together (it may valid asset-type specific information - e.g. that
41+
// there are no classes in shared library filenames).
42+
typedef ApplicationAssetValidator = Future<ValidationErrors> Function(
43+
List<EncodedAsset> assets);
44+
2645
/// The programmatic API to be used by Dart launchers to invoke native builds.
2746
///
2847
/// These methods are invoked by launchers such as dartdev (for `dart run`)
@@ -58,6 +77,8 @@ class NativeAssetsBuildRunner {
5877
required Target target,
5978
required Uri workingDirectory,
6079
required BuildMode buildMode,
80+
required BuildValidator buildValidator,
81+
required ApplicationAssetValidator applicationAssetValidator,
6182
CCompilerConfig? cCompilerConfig,
6283
IOSSdk? targetIOSSdk,
6384
int? targetIOSVersion,
@@ -66,10 +87,14 @@ class NativeAssetsBuildRunner {
6687
required bool includeParentEnvironment,
6788
PackageLayout? packageLayout,
6889
String? runPackageName,
69-
Iterable<String>? supportedAssetTypes,
90+
required Iterable<String> supportedAssetTypes,
7091
required bool linkingEnabled,
7192
}) async =>
7293
_run(
94+
validator: (HookConfig config, HookOutputImpl output) =>
95+
buildValidator(config as BuildConfig, output as BuildOutput),
96+
applicationAssetValidator: (assets) async =>
97+
linkingEnabled ? [] : applicationAssetValidator(assets),
7398
hook: Hook.build,
7499
linkModePreference: linkModePreference,
75100
target: target,
@@ -103,6 +128,8 @@ class NativeAssetsBuildRunner {
103128
required Target target,
104129
required Uri workingDirectory,
105130
required BuildMode buildMode,
131+
required LinkValidator linkValidator,
132+
required ApplicationAssetValidator applicationAssetValidator,
106133
CCompilerConfig? cCompilerConfig,
107134
IOSSdk? targetIOSSdk,
108135
int? targetIOSVersion,
@@ -112,10 +139,14 @@ class NativeAssetsBuildRunner {
112139
PackageLayout? packageLayout,
113140
Uri? resourceIdentifiers,
114141
String? runPackageName,
115-
Iterable<String>? supportedAssetTypes,
142+
required Iterable<String> supportedAssetTypes,
116143
required BuildResult buildResult,
117144
}) async =>
118145
_run(
146+
validator: (HookConfig config, HookOutputImpl output) =>
147+
linkValidator(config as LinkConfig, output as LinkOutput),
148+
applicationAssetValidator: (assets) async =>
149+
applicationAssetValidator(assets),
119150
hook: Hook.link,
120151
linkModePreference: linkModePreference,
121152
target: target,
@@ -141,6 +172,8 @@ class NativeAssetsBuildRunner {
141172
required Target target,
142173
required Uri workingDirectory,
143174
required BuildMode buildMode,
175+
required _HookValidator validator,
176+
required ApplicationAssetValidator applicationAssetValidator,
144177
CCompilerConfig? cCompilerConfig,
145178
IOSSdk? targetIOSSdk,
146179
int? targetIOSVersion,
@@ -150,7 +183,7 @@ class NativeAssetsBuildRunner {
150183
PackageLayout? packageLayout,
151184
Uri? resourceIdentifiers,
152185
String? runPackageName,
153-
Iterable<String>? supportedAssetTypes,
186+
required Iterable<String> supportedAssetTypes,
154187
BuildResult? buildResult,
155188
bool? linkingEnabled,
156189
}) async {
@@ -236,6 +269,7 @@ class NativeAssetsBuildRunner {
236269
final (hookOutput, packageSuccess) = await _runHookForPackageCached(
237270
hook,
238271
config,
272+
validator,
239273
packageLayout.packageConfigUri,
240274
workingDirectory,
241275
includeParentEnvironment,
@@ -246,17 +280,14 @@ class NativeAssetsBuildRunner {
246280
metadata[config.packageName] = hookOutput.metadata;
247281
}
248282

249-
// Note the caller will need to check whether there are no duplicates
250-
// between the build and link hook.
251-
final validateResult = validateNoDuplicateDylibs(hookResult.assets);
252-
if (validateResult.isNotEmpty) {
253-
for (final error in validateResult) {
254-
logger.severe(error);
255-
}
256-
hookResult = hookResult.copyAdd(HookOutputImpl(), false);
257-
}
283+
final errors = await applicationAssetValidator(hookResult.encodedAssets);
284+
if (errors.isEmpty) return hookResult;
258285

259-
return hookResult;
286+
logger.severe('Application asset verification failed:');
287+
for (final error in errors) {
288+
logger.severe('- $error');
289+
}
290+
return HookResult.failure();
260291
}
261292

262293
static Future<HookConfigImpl> _cliConfig(
@@ -272,7 +303,7 @@ class NativeAssetsBuildRunner {
272303
int? targetAndroidNdkApi,
273304
int? targetIOSVersion,
274305
int? targetMacOSVersion,
275-
Iterable<String>? supportedAssetTypes,
306+
Iterable<String> supportedAssetTypes,
276307
Hook hook,
277308
Uri? resourceIdentifiers,
278309
BuildResult? buildResult,
@@ -331,7 +362,7 @@ class NativeAssetsBuildRunner {
331362
cCompiler: cCompilerConfig,
332363
targetAndroidNdkApi: targetAndroidNdkApi,
333364
recordedUsagesFile: resourcesFile?.uri,
334-
assets: buildResult!.assetsForLinking[package.name] ?? [],
365+
encodedAssets: buildResult!.encodedAssetsForLinking[package.name] ?? [],
335366
supportedAssetTypes: supportedAssetTypes,
336367
linkModePreference: linkModePreference,
337368
);
@@ -370,9 +401,10 @@ class NativeAssetsBuildRunner {
370401
required Uri workingDirectory,
371402
required bool includeParentEnvironment,
372403
required bool linkingEnabled,
404+
required BuildValidator buildValidator,
373405
PackageLayout? packageLayout,
374406
String? runPackageName,
375-
Iterable<String>? supportedAssetTypes,
407+
required Iterable<String> supportedAssetTypes,
376408
}) async {
377409
const hook = Hook.build;
378410
packageLayout ??= await PackageLayout.fromRootPackageRoot(workingDirectory);
@@ -413,7 +445,7 @@ class NativeAssetsBuildRunner {
413445
continue;
414446
}
415447
// TODO(https://github.com/dart-lang/native/issues/1321): Should dry runs be cached?
416-
var (buildOutput, packageSuccess) = await runUnderDirectoriesLock(
448+
final (buildOutput, packageSuccess) = await runUnderDirectoriesLock(
417449
[
418450
Directory.fromUri(config.outputDirectoryShared.parent),
419451
Directory.fromUri(config.outputDirectory.parent),
@@ -423,6 +455,8 @@ class NativeAssetsBuildRunner {
423455
() => _runHookForPackage(
424456
hook,
425457
config,
458+
(HookConfig config, HookOutputImpl output) =>
459+
buildValidator(config as BuildConfig, output as BuildOutput),
426460
packageConfigUri,
427461
workingDirectory,
428462
includeParentEnvironment,
@@ -431,38 +465,15 @@ class NativeAssetsBuildRunner {
431465
packageLayout!,
432466
),
433467
);
434-
buildOutput = _expandArchsCodeAssets(buildOutput);
435468
hookResult = hookResult.copyAdd(buildOutput, packageSuccess);
436469
}
437470
return hookResult;
438471
}
439472

440-
HookOutputImpl _expandArchsCodeAssets(HookOutputImpl buildOutput) {
441-
final assets = <Asset>[];
442-
for (final asset in buildOutput.assets) {
443-
switch (asset) {
444-
case CodeAsset _:
445-
if (asset.architecture != null) {
446-
// Backwards compatibility, if an architecture is provided use it.
447-
assets.add(asset);
448-
} else {
449-
// Dry run does not report architecture. Dart VM branches on OS
450-
// and Target when looking up assets, so populate assets for all
451-
// architectures.
452-
for (final architecture in asset.os.architectures) {
453-
assets.add(asset.copyWith(architecture: architecture));
454-
}
455-
}
456-
case DataAsset _:
457-
assets.add(asset);
458-
}
459-
}
460-
return buildOutput.copyWith(assets: assets);
461-
}
462-
463473
Future<_PackageBuildRecord> _runHookForPackageCached(
464474
Hook hook,
465475
HookConfigImpl config,
476+
_HookValidator validator,
466477
Uri packageConfigUri,
467478
Uri workingDirectory,
468479
bool includeParentEnvironment,
@@ -497,25 +508,13 @@ class NativeAssetsBuildRunner {
497508
final lastBuilt = hookOutput.timestamp.roundDownToSeconds();
498509
final dependenciesLastChange =
499510
await hookOutput.dependenciesModel.lastModified();
500-
late final DateTime assetsLastChange;
501-
if (hook == Hook.link) {
502-
assetsLastChange = await (config as LinkConfigImpl)
503-
.assets
504-
.map((a) => a.file)
505-
.whereType<Uri>()
506-
.map((u) => u.fileSystemEntity)
507-
.lastModified();
508-
}
509511
if (lastBuilt.isAfter(dependenciesLastChange) &&
510-
lastBuilt.isAfter(hookLastSourceChange) &&
511-
(hook == Hook.build || lastBuilt.isAfter(assetsLastChange))) {
512+
lastBuilt.isAfter(hookLastSourceChange)) {
512513
logger.info(
513514
[
514515
'Skipping ${hook.name} for ${config.packageName} in $outDir.',
515516
'Last build on $lastBuilt.',
516517
'Last dependencies change on $dependenciesLastChange.',
517-
if (hook == Hook.link)
518-
'Last assets for linking change on $assetsLastChange.',
519518
'Last hook change on $hookLastSourceChange.',
520519
].join(' '),
521520
);
@@ -528,6 +527,7 @@ class NativeAssetsBuildRunner {
528527
return await _runHookForPackage(
529528
hook,
530529
config,
530+
validator,
531531
packageConfigUri,
532532
workingDirectory,
533533
includeParentEnvironment,
@@ -542,6 +542,7 @@ class NativeAssetsBuildRunner {
542542
Future<_PackageBuildRecord> _runHookForPackage(
543543
Hook hook,
544544
HookConfigImpl config,
545+
_HookValidator validator,
545546
Uri packageConfigUri,
546547
Uri workingDirectory,
547548
bool includeParentEnvironment,
@@ -601,16 +602,12 @@ ${result.stdout}
601602
final output = HookOutputImpl.readFromFile(file: config.outputFile) ??
602603
HookOutputImpl();
603604

604-
final validateResult = await validate(
605-
config,
606-
output,
607-
packageLayout,
608-
);
609-
success &= validateResult.success;
610-
if (!validateResult.success) {
605+
final errors = await _validate(config, output, packageLayout, validator);
606+
success &= errors.isEmpty;
607+
if (errors.isNotEmpty) {
611608
logger.severe('package:${config.packageName}` has invalid output.');
612609
}
613-
for (final error in validateResult.errors) {
610+
for (final error in errors) {
614611
logger.severe('- $error');
615612
}
616613
return (output, success);
@@ -758,7 +755,7 @@ ${compileResult.stdout}
758755
required OS targetOS,
759756
required LinkModePreference linkMode,
760757
required Uri buildParentDir,
761-
Iterable<String>? supportedAssetTypes,
758+
required Iterable<String> supportedAssetTypes,
762759
required bool? linkingEnabled,
763760
}) async {
764761
const hook = Hook.build;
@@ -814,32 +811,33 @@ ${compileResult.stdout}
814811
};
815812
}
816813

817-
Future<ValidateResult> validate(
814+
Future<ValidationErrors> _validate(
818815
HookConfigImpl config,
819816
HookOutputImpl output,
820817
PackageLayout packageLayout,
818+
_HookValidator validator,
821819
) async {
822-
var (errors: errors, success: success) = config is BuildConfigImpl
823-
? await validateBuild(config, output)
824-
: await validateLink(config as LinkConfigImpl, output);
820+
final errors = config is BuildConfigImpl
821+
? await validateBuildOutput(config, output)
822+
: await validateLinkOutput(config as LinkConfig, output);
823+
errors.addAll(await validator(config, output));
825824

826825
if (config is BuildConfigImpl) {
827826
final packagesWithLink =
828827
(await packageLayout.packagesWithAssets(Hook.link))
829828
.map((p) => p.name);
830-
for (final targetPackage in output.assetsForLinking.keys) {
829+
for (final targetPackage in output.encodedAssetsForLinking.keys) {
831830
if (!packagesWithLink.contains(targetPackage)) {
832-
for (final asset in output.assetsForLinking[targetPackage]!) {
833-
success &= false;
831+
for (final asset in output.encodedAssetsForLinking[targetPackage]!) {
834832
errors.add(
835-
'Asset "${asset.id}" is sent to package "$targetPackage" for'
833+
'Asset "$asset" is sent to package "$targetPackage" for'
836834
' linking, but that package does not have a link hook.',
837835
);
838836
}
839837
}
840838
}
841839
}
842-
return (errors: errors, success: success);
840+
return errors;
843841
}
844842

845843
Future<(List<Package> plan, PackageGraph? dependencyGraph, bool success)>
@@ -878,9 +876,9 @@ ${compileResult.stdout}
878876
// Link hooks are skipped if no assets for linking are provided.
879877
buildPlan = [];
880878
final skipped = <String>[];
881-
final assetsForLinking = buildResult?.assetsForLinking;
879+
final encodedAssetsForLinking = buildResult?.encodedAssetsForLinking;
882880
for (final package in packagesWithHook) {
883-
if (assetsForLinking![package.name]?.isNotEmpty ?? false) {
881+
if (encodedAssetsForLinking![package.name]?.isNotEmpty ?? false) {
884882
buildPlan.add(package);
885883
} else {
886884
skipped.add(package.name);

pkgs/native_assets_builder/lib/src/model/build_dry_run_result.dart

+2-2
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,13 @@ import '../../native_assets_builder.dart';
1010
/// the dependency tree of the entry point application.
1111
abstract interface class BuildDryRunResult {
1212
/// The native assets produced by the hooks, which should be bundled.
13-
List<Asset> get assets;
13+
List<EncodedAsset> get encodedAssets;
1414

1515
/// Whether all hooks completed without errors.
1616
///
1717
/// All error messages are streamed to [NativeAssetsBuildRunner.logger].
1818
bool get success;
1919

2020
/// The native assets produced by the hooks, which should be linked.
21-
Map<String, List<Asset>> get assetsForLinking;
21+
Map<String, List<EncodedAsset>> get encodedAssetsForLinking;
2222
}

pkgs/native_assets_builder/lib/src/model/build_result.dart

+2-2
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ abstract class BuildResult {
1818
bool get success;
1919

2020
/// The native assets produced by the hooks, which should be bundled.
21-
List<Asset> get assets;
21+
List<EncodedAsset> get encodedAssets;
2222

2323
/// The native assets produced by the hooks, which should be linked.
24-
Map<String, List<Asset>> get assetsForLinking;
24+
Map<String, List<EncodedAsset>> get encodedAssetsForLinking;
2525
}

0 commit comments

Comments
 (0)