-
Notifications
You must be signed in to change notification settings - Fork 8
Internal precompile binaries hook #22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
BitcoinZavior
wants to merge
42
commits into
bitcoindevkit:main
Choose a base branch
from
LtbLightning:internal-precompile-binaries-hook
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
42 commits
Select commit
Hold shift + click to select a range
b7fe450
Add core utility functions for hex encoding and HTTP retry
BitcoinZavior 85dfab4
Add utility functions and target triple mapping
BitcoinZavior feb7f85
Add options and configuration parsing
BitcoinZavior e126921
Add crate hash calculation
BitcoinZavior 9d582c6
Add Rust toolchain detection
BitcoinZavior 1e2df77
Add Cargo build integration
BitcoinZavior 6c6f974
Add artifacts provider for download and verification
BitcoinZavior c19cef8
Add precompiled builder main integration
BitcoinZavior 7dab321
Add CLI support utilities
BitcoinZavior 224d288
Add CLI framework
BitcoinZavior bb8ca56
Add CLI commands: targets and hash
BitcoinZavior ac5f174
Add CLI commands: gen_key and sign
BitcoinZavior 8434ef4
Add CLI command: precompile_binaries
BitcoinZavior e2781fe
Add build tool executable
BitcoinZavior 0d197a2
Add tests for precompiled binaries
BitcoinZavior d7d4a3e
Update artifact host references in tests
BitcoinZavior 675d6e9
Add documentation for precompiled binaries
BitcoinZavior e9051e6
Add CI workflow for precompiling binaries
BitcoinZavior 518cfa5
Integrate precompile workflow into main CI
BitcoinZavior 7728dda
Update build hook to use precompiled builder
BitcoinZavior c062e86
Format crate_hash.dart code
BitcoinZavior 8576a9c
Add dependencies and configuration to pubspec.yaml
BitcoinZavior fbf3f6f
Update .gitignore and README documentation
BitcoinZavior e73cf33
Update artifact host to bitcoindevkit/bdk-dart
BitcoinZavior 8ed8ef6
Format rust_toolchain.dart code
BitcoinZavior 5edfa04
Format crate_hash_test.dart code
BitcoinZavior c572e77
Remove trailing newlines from CLI files
BitcoinZavior 0087a44
Remove trailing newline from target.dart
BitcoinZavior 4e3094e
update readme
BitcoinZavior 00b0daf
Pin Rust toolchain to 1.85.1 in CI workflow
BitcoinZavior 8412bc8
Add exception class for required precompiled binaries
BitcoinZavior 1f98a55
Update precompiled binaries documentation intro
BitcoinZavior f6ee437
Add mode behavior documentation section
BitcoinZavior c1446c1
Update README precompiled binaries intro
BitcoinZavior cdc2fc2
Update README mode behavior descriptions
BitcoinZavior 229cdeb
Add repository parameter to GitHub CLI commands
BitcoinZavior 8a8e82d
update readme
BitcoinZavior 54394f2
docs: state that always mode throws on failure
BitcoinZavior fcff476
docs: trim trailing newline in example
BitcoinZavior d137ff9
style: format PrecompiledBinaryRequiredException.toString
BitcoinZavior 16ac553
fix: correct GitHub release create argument order
BitcoinZavior b55cbce
chore: add native Cargo.lock
BitcoinZavior File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| name: Precompile Binaries | ||
|
|
||
| on: | ||
| workflow_call: | ||
| secrets: | ||
| PRECOMPILED_PRIVATE_KEY: | ||
| required: true | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
|
|
||
| env: | ||
| CRATE_DIR: "native" | ||
| CRATE_PACKAGE: "bdk_dart_ffi" | ||
|
|
||
| jobs: | ||
| precompile_macos_ios: | ||
| runs-on: macos-latest | ||
| env: | ||
| PRIVATE_KEY: ${{ secrets.PRECOMPILED_PRIVATE_KEY }} | ||
| GH_TOKEN: ${{ github.token }} | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions-rs/toolchain@v1 | ||
reez marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| with: | ||
| profile: minimal | ||
| toolchain: 1.85.1 | ||
| override: true | ||
| - uses: dart-lang/setup-dart@v1 | ||
| - uses: subosito/flutter-action@v2 | ||
| with: | ||
| channel: "stable" | ||
| - name: Pub get | ||
| run: dart pub get | ||
| - name: Precompile (macOS + iOS) | ||
| run: | | ||
| set -euo pipefail | ||
| dart run bin/build_tool.dart precompile-binaries \ | ||
| -v \ | ||
| --os=macos \ | ||
| --manifest-dir="${CRATE_DIR}" \ | ||
| --crate-package="${CRATE_PACKAGE}" \ | ||
| --repository="${GITHUB_REPOSITORY}" | ||
|
|
||
| precompile_android: | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| PRIVATE_KEY: ${{ secrets.PRECOMPILED_PRIVATE_KEY }} | ||
| GH_TOKEN: ${{ github.token }} | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| ANDROID_NDK_VERSION: "26.3.11579264" | ||
| ANDROID_MIN_SDK: "23" | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions-rs/toolchain@v1 | ||
| with: | ||
| profile: minimal | ||
| toolchain: 1.85.1 | ||
| override: true | ||
| - uses: dart-lang/setup-dart@v1 | ||
| - name: Set up Android SDK | ||
| uses: android-actions/setup-android@v3 | ||
| - name: Install NDK | ||
| run: | | ||
| set -euo pipefail | ||
| sdkmanager --install "ndk;${ANDROID_NDK_VERSION}" | ||
| - name: Install cargo-ndk | ||
| run: cargo install cargo-ndk --locked | ||
| - name: Pub get | ||
| run: dart pub get | ||
| - name: Precompile (Android) | ||
| env: | ||
| ANDROID_SDK_ROOT: ${{ env.ANDROID_SDK_ROOT }} | ||
| run: | | ||
| set -euo pipefail | ||
| ANDROID_SDK_ROOT="${ANDROID_SDK_ROOT:-$ANDROID_HOME}" | ||
| dart run bin/build_tool.dart precompile-binaries \ | ||
| -v \ | ||
| --os=android \ | ||
| --manifest-dir="${CRATE_DIR}" \ | ||
| --crate-package="${CRATE_PACKAGE}" \ | ||
| --repository="${GITHUB_REPOSITORY}" \ | ||
| --android-sdk-location="${ANDROID_SDK_ROOT}" \ | ||
| --android-ndk-version="${ANDROID_NDK_VERSION}" \ | ||
| --android-min-sdk-version="${ANDROID_MIN_SDK}" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,6 @@ | |
| /target | ||
| native/target/ | ||
| **/*.rs.bk | ||
| Cargo.lock | ||
|
|
||
| # Flutter/Dart | ||
| **/build/ | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| import 'package:bdk_dart/src/precompiled/cli/cli.dart'; | ||
|
|
||
| Future<void> main(List<String> args) async { | ||
| await runCli(args); | ||
| } | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Precompiled binaries (maintainers) | ||
|
|
||
| This document describes how precompiled binaries are built, signed, and published for the plugin. | ||
|
|
||
| ## Overview | ||
|
|
||
| - CI builds and uploads precompiled binaries via `.github/workflows/precompile_binaries.yml`. | ||
| - Artifacts are tagged by the crate hash and uploaded to a GitHub release. | ||
| - Each binary is signed with an Ed25519 key; the public key is embedded in `pubspec.yaml`. | ||
| - The build hook downloads verified binaries when appropriate (depending on mode configuration) and falls back to local builds if needed. | ||
|
|
||
| ## Mode behavior | ||
|
|
||
| The `mode` configuration in `pubspec.yaml` controls fallback behavior: | ||
|
|
||
| - `auto`: Uses a heuristic to prefer local builds for development. If the Rust toolchain (`rustup`) is detected, it disables precompiled binaries and builds locally. If no Rust toolchain is found, it uses precompiled binaries. This provides optimal developer experience while keeping end-user builds fast. | ||
| - `always`: Throws an exception if download/verification fails; does not fall back. | ||
| - `never`: Always builds locally via the standard build hook, ignoring precompiled binaries. | ||
|
|
||
| ## CI workflow | ||
|
|
||
| The workflow runs on `push` to `main` and on manual dispatch. It invokes: | ||
|
|
||
| ``` | ||
| dart run bin/build_tool.dart precompile-binaries ... | ||
| ``` | ||
|
|
||
| It currently builds macOS/iOS and Android targets. | ||
|
|
||
| ## Release expectations | ||
|
|
||
| - The workflow creates/releases a GitHub release named `precompiled_<crateHash>` where `<crateHash>` comes from the verified crate sources and config. | ||
| - If the release already exists, the workflow uploads missing assets without rebuilding anything already present. | ||
| - If `gh release view precompiled_<crateHash>` fails locally, rerun `dart run bin/build_tool.dart precompile-binaries ...` with the same crate hash to recreate or update the release. | ||
|
|
||
| ## How the download works | ||
|
|
||
| - The crate hash is computed from the Rust crate sources plus the plugin's `precompiled_binaries` config. | ||
| - The release tag is `precompiled_<crateHash>`. | ||
| - Assets are named `<targetTriple>_<libraryFileName>` with a matching `.sig` file. | ||
| - Each binary is paired with the `.sig` file that the hook uses to verify the download before applying it. | ||
| - The hook chooses the correct `lib$cratePackage` (or `lib$cratePackage.so`) artifact by matching the target triple and link mode from the Dart build config. | ||
| - On build, the hook downloads the signature and binary, verifies it, then places it in the build output. | ||
| - If any step fails (missing asset, bad signature), the hook builds locally via the standard build hook. | ||
|
|
||
| ## Manual release (local) | ||
|
|
||
| Use this when debugging CI or producing artifacts manually. | ||
|
|
||
| Required environment variables: | ||
|
|
||
| - `PRIVATE_KEY` (Ed25519 private key, hex-encoded, 64 bytes) | ||
| - `GH_TOKEN` or `GITHUB_TOKEN` (GitHub token with release upload permissions) | ||
|
|
||
| Example: | ||
|
|
||
| ``` | ||
| dart run bin/build_tool.dart precompile-binaries \ | ||
| --manifest-dir="native" \ | ||
| --crate-package="bdk_dart_ffi" \ | ||
| --repository="owner/repo" \ | ||
| --os=macos | ||
| ``` | ||
|
|
||
| ## Troubleshooting & ops tips | ||
|
|
||
| - If `gh release view precompiled_<crateHash>` shows a release without the expected `<targetTriple>_` assets, rerun the build locally to regenerate them. | ||
| - A stale crate hash (because sources or `precompiled_binaries` config changed) will point to a release that either doesn’t exist yet or lacks current binaries; re-run `dart run bin/build_tool.dart hash --manifest-dir=native` to confirm the hash and rebuild with the same inputs. | ||
| - Use `gh release view precompiled_<crateHash> --json assets --jq '.assets[].name'` to inspect what’s uploaded and verify `.sig` coverage. | ||
| - When debugging download failures, set `BDK_DART_PRECOMPILED_VERBOSE=1` to see why the hook skipped an asset. | ||
|
|
||
| ## Configuration knobs | ||
|
|
||
| - `rust-toolchain.toml` controls the Rust channel and target list. | ||
| - `pubspec.yaml` under `bdk_dart.precompiled_binaries` must include: | ||
| - `artifact_host` (owner/repo) | ||
| - `public_key` (Ed25519 public key, hex-encoded, 32 bytes) | ||
|
|
||
| ## Environment, keys, and secrets | ||
|
|
||
| - `PRIVATE_KEY`: 64-byte hex string (Ed25519 private key). This must be set locally or as a GitHub Actions secret before running `precompile-binaries`. Keep it out of source control. | ||
| - `PUBLIC_KEY`: Add the matching 32-byte hex public key to `pubspec.yaml` so consumers can verify downloads. | ||
| - `GH_TOKEN` / `GITHUB_TOKEN`: release upload permissions (already used in the CI workflow). | ||
| - `BDK_DART_PRECOMPILED_VERBOSE=1`: optional; shows download and verification details when debugging consumer builds. | ||
|
|
||
| Generate a keypair with `dart run bin/build_tool.dart gen-key` and copy the printed `PRIVATE_KEY`/`PUBLIC_KEY` values. Rotate the pair if you ever suspect the signing key was exposed, and update every release’s config accordingly. | ||
|
|
||
| ## Security reminder | ||
|
|
||
| - Treat the `PRIVATE_KEY` used for signing as highly sensitive; do not commit it to version control and rotate it immediately if you suspect compromise. | ||
| - Update the public key in `pubspec.yaml` if the private key is rotated so consumers can still verify downloads. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,25 @@ | ||
| import 'package:bdk_dart/src/precompiled/precompiled_builder.dart'; | ||
| import 'package:hooks/hooks.dart'; | ||
| import 'package:native_toolchain_rust/native_toolchain_rust.dart'; | ||
| import 'package:native_toolchain_rust/native_toolchain_rust.dart' as ntr; | ||
|
|
||
| void main(List<String> args) async { | ||
| await build(args, (input, output) async { | ||
| await const RustBuilder( | ||
| final builder = PrecompiledBuilder( | ||
| assetName: 'uniffi:bdk_dart_ffi', | ||
| ).run(input: input, output: output); | ||
| buildModeName: ntr.BuildMode.release.name, | ||
| fallback: (input, output, assetRouting, logger) async { | ||
| final rustBuilder = ntr.RustBuilder( | ||
| assetName: 'uniffi:bdk_dart_ffi', | ||
| buildMode: ntr.BuildMode.release, | ||
| ); | ||
| await rustBuilder.run( | ||
| input: input, | ||
| output: output, | ||
| assetRouting: assetRouting, | ||
| logger: logger, | ||
| ); | ||
| }, | ||
| ); | ||
| await builder.run(input: input, output: output); | ||
| }); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.