From 38206afdbb61d023905da51804ac867f4183ef7b Mon Sep 17 00:00:00 2001 From: Walter Perdan Date: Fri, 17 Jul 2026 20:36:09 +0200 Subject: [PATCH] doc(wasm): add JS/TS integration guide + link from README (#226) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The npm package (@webarkit/webarkitlib-wasm) shipped with no consumer-facing usage guide — the README only said "see npm for API docs". Add a full walkthrough at docs/wasm-js-integration.md covering: - loading + initializing the module (init / init_wasm / init_panic_hook) - the standard vs SIMD engines (the package `exports` map) + a runtime probe - the three handles: WasmARHandle (square markers), WasmKpmHandle (NFT detection), WasmNFTHandle (AR2 tracking) - the NFT detect-then-track loop with the real serialized shapes ({ pose, page, error } and { found, matrix, error, cont_num }) - the 3x4 row-major pose-matrix layout, TypeScript usage, memory management (free()), and common pitfalls (RGBA length, load-before-detect, etc.) All snippets are checked against the actual crates/wasm/src/lib.rs API. Link from the README's WASM section using an absolute GitHub URL so it also works for consumers reading the README on npmjs.com (only README.md is published to the package, not docs/). The root README is the source copied to pkg/README.md by build-dual at build time; pkg/README.md refreshes on the next `npm run build`. Second slice of the #226 documentation pass (after the KPM rustdoc slice in #231). Refs #226 Co-Authored-By: Claude Opus 4.8 --- README.md | 4 +- docs/wasm-js-integration.md | 275 ++++++++++++++++++++++++++++++++++++ 2 files changed, 278 insertions(+), 1 deletion(-) create mode 100644 docs/wasm-js-integration.md diff --git a/README.md b/README.md index 2e08be4..785f431 100644 --- a/README.md +++ b/README.md @@ -209,7 +209,9 @@ For integration into your own project, install the pre-built package from npm npm install @webarkit/webarkitlib-wasm ``` -See [@webarkit/webarkitlib-wasm](https://www.npmjs.com/package/@webarkit/webarkitlib-wasm) for API documentation and usage examples. +📖 **[WebAssembly (JS/TS) integration guide](https://github.com/webarkit/WebARKitLib-rs/blob/main/docs/wasm-js-integration.md)** — a full walkthrough: loading and initializing the module, the standard vs SIMD engines, the three handles (`WasmARHandle` / `WasmKpmHandle` / `WasmNFTHandle`), the NFT detect-then-track loop, the pose-matrix layout, TypeScript usage, and memory management. + +See also [@webarkit/webarkitlib-wasm](https://www.npmjs.com/package/@webarkit/webarkitlib-wasm) on npm. #### Running the bundled demos diff --git a/docs/wasm-js-integration.md b/docs/wasm-js-integration.md new file mode 100644 index 0000000..b73d9a4 --- /dev/null +++ b/docs/wasm-js-integration.md @@ -0,0 +1,275 @@ +# WebAssembly (JS/TS) integration guide + +How to use [`@webarkit/webarkitlib-wasm`](https://www.npmjs.com/package/@webarkit/webarkitlib-wasm) +— the WebAssembly build of WebARKitLib-rs — from a browser app. The whole +pipeline runs the **pure-Rust** matcher; there is no C++ or Emscripten +dependency at runtime. + +- [Install](#install) +- [Load and initialize the module](#load-and-initialize-the-module) +- [Standard vs SIMD build](#standard-vs-simd-build) +- [The three handles](#the-three-handles) +- [NFT: KPM detection → AR2 tracking](#nft-kpm-detection--ar2-tracking) + - [1. Fetch the assets](#1-fetch-the-assets) + - [2. Detect the initial pose (KPM)](#2-detect-the-initial-pose-kpm) + - [3. Refine per frame (AR2 tracking)](#3-refine-per-frame-ar2-tracking) + - [4. Full detect-then-track loop](#4-full-detect-then-track-loop) +- [Square marker detection](#square-marker-detection) +- [The pose matrix](#the-pose-matrix) +- [TypeScript](#typescript) +- [Memory management](#memory-management) +- [Common pitfalls](#common-pitfalls) + +## Install + +```bash +npm install @webarkit/webarkitlib-wasm +``` + +The package ships two pre-built engines and no build step is required. It is +an ES module (`"type": "module"`). + +## Load and initialize the module + +wasm-bindgen produces a default-export `init` function that fetches and +instantiates the `.wasm`. Call it once, then call `init_wasm()` to install the +console logger, before constructing any handle: + +```js +import init, { init_wasm, get_version } from '@webarkit/webarkitlib-wasm'; + +await init(); // instantiate the wasm module +init_wasm(); // wire up console logging (optional but recommended) +console.log(get_version()); +``` + +If you want Rust panics to surface as readable console errors during +development, also call `init_panic_hook()` after `init()`. + +## Standard vs SIMD build + +The package exposes two engines through its `exports` map: + +| Import | Engine | When | +|--------|--------|------| +| `@webarkit/webarkitlib-wasm` | Standard (portable) | Default — works everywhere | +| `@webarkit/webarkitlib-wasm/simd` | WASM SIMD | Faster; needs a browser with WASM SIMD | + +```js +// Standard +import init, { WasmKpmHandle } from '@webarkit/webarkitlib-wasm'; +// SIMD (identical API) +import init, { WasmKpmHandle } from '@webarkit/webarkitlib-wasm/simd'; +``` + +The two engines expose the **same API**, so you can feature-detect and pick +one at runtime: + +```js +// Minimal WASM-SIMD probe (the canonical 8-byte "has SIMD" module). +const simdOk = WebAssembly.validate(new Uint8Array([ + 0,97,115,109,1,0,0,0,1,5,1,96,0,1,123,3,2,1,0,10,10,1,8,0,65,0,253,15,253,98,11, +])); +const mod = simdOk + ? await import('@webarkit/webarkitlib-wasm/simd') + : await import('@webarkit/webarkitlib-wasm'); +await mod.default(); +mod.init_wasm(); +``` + +## The three handles + +| Handle | Purpose | Input marker data | +|--------|---------|-------------------| +| `WasmARHandle` | Square **pattern / barcode** marker detection + pose | `.patt` file, or matrix-code type | +| `WasmKpmHandle` | **NFT detection** — finds a natural-feature marker and returns its initial pose | `.fset3` | +| `WasmNFTHandle` | **NFT tracking** — refines a pose frame-to-frame via AR2 template matching | `.iset` + `.fset` | + +For NFT you use `WasmKpmHandle` and `WasmNFTHandle` together: KPM finds the +marker cold, AR2 tracking keeps it locked cheaply once found. This mirrors the +native `simple_nft` example. + +## NFT: KPM detection → AR2 tracking + +### 1. Fetch the assets + +You need the camera parameters and the marker set produced by +`nft_marker_gen` (`.fset3` + `.iset` + `.fset`). Load each as a `Uint8Array`: + +```js +async function bytes(url) { + const res = await fetch(url); + return new Uint8Array(await res.arrayBuffer()); +} + +const cameraParam = await bytes('assets/camera_para.dat'); +const fset3 = await bytes('assets/pinball.fset3'); +const iset = await bytes('assets/pinball.iset'); +const fset = await bytes('assets/pinball.fset'); +``` + +### 2. Detect the initial pose (KPM) + +Construct with the camera parameters and the **frame size** in pixels. The +constructor scales the camera intrinsics to that size for you (the +`arParamChangeSize` equivalent), so pass the dimensions of the frames you will +feed to `detect()`. + +```js +import { WasmKpmHandle } from '@webarkit/webarkitlib-wasm'; + +const width = 640, height = 480; +const kpm = new WasmKpmHandle(cameraParam, width, height); +kpm.load_ref_data(fset3); // KPM reference keypoints + +// `rgba` is a width*height*4 buffer, e.g. from a canvas: +// ctx.getImageData(0, 0, width, height).data +const result = kpm.detect(rgba); +if (result) { + // result: { pose: number[12], page: number, error: number } + console.log('found page', result.page, 'error', result.error); +} +``` + +`detect()` returns `null` when no marker is found, or +`{ pose, page, error }` where `pose` is the 3×4 camera pose flattened to 12 +row-major floats. See [The pose matrix](#the-pose-matrix). + +### 3. Refine per frame (AR2 tracking) + +Feed the KPM pose into a `WasmNFTHandle` as the initial pose, then call +`track()` each frame: + +```js +import { WasmNFTHandle } from '@webarkit/webarkitlib-wasm'; + +const nft = new WasmNFTHandle(cameraParam, width, height); +nft.load_nft_marker(iset, fset); +nft.set_initial_pose(result.pose); // Float32Array or number[12] from KPM + +const tracked = nft.track(rgba, width, height); +if (tracked.found) { + // tracked: { found, matrix: number[12], error, cont_num } + render(tracked.matrix); +} +``` + +### 4. Full detect-then-track loop + +The usual pattern: run cheap AR2 tracking while it holds, and fall back to KPM +detection when tracking is lost (`found === false`, or `cont_num` resets to 0). + +```js +let locked = false; + +function onFrame(rgba) { + if (locked) { + const t = nft.track(rgba, width, height); + if (t.found) { render(t.matrix); return; } + locked = false; // lost it — re-detect next frame + nft.reset_tracking(); + } + + const d = kpm.detect(rgba); // cold detect + if (d) { + nft.set_initial_pose(d.pose); + locked = true; + render(d.pose); + } +} +``` + +## Square marker detection + +For classic ARToolKit square markers (Hiro pattern, barcodes) use +`WasmARHandle` instead: + +```js +import { WasmARHandle } from '@webarkit/webarkitlib-wasm'; + +const ar = new WasmARHandle(cameraParam); // frame size taken from param +const pattId = ar.load_pattern(await (await fetch('assets/patt.hiro')).text()); +ar.set_threshold(100); + +const markers = ar.detect_markers(rgba, width, height, /* ...pixel format */); +// then ar.get_trans_mat(markerIdx, markerWidthMm) for a pose +``` + +`WasmARHandle` also supports matrix-code (barcode) detection via +`set_matrix_code_type(...)` — see the `simple_video_marker_example.html` demo. + +## The pose matrix + +Both `detect()` and `track()` return the pose as **12 floats = a 3×4 +row-major camera pose matrix** `[R | t]`: + +```text +[ m0 m1 m2 m3 ] indices: [ 0 1 2 3 ] +[ m4 m5 m6 m7 ] [ 4 5 6 7 ] +[ m8 m9 m10 m11] [ 8 9 10 11] +``` + +The upper-left 3×3 is rotation; the last column is translation in +millimetres. To drive a WebGL/Three.js/Babylon renderer, expand it to a 4×4 +model-view matrix (append `[0,0,0,1]`) and apply your library's coordinate +convention. This is the same matrix layout the native API and jsartoolkitNFT +use, so existing conversion helpers apply. + +## TypeScript + +Type definitions ship with the package +(`dist-std/webarkitlib_wasm.d.ts`, referenced by `"types"` in +`package.json`), so imports are typed out of the box. The methods that return +serialized objects (`detect`, `track`, `get_trans_mat`) are typed as `any` +because they cross the wasm boundary as `JsValue`. Declare the shapes yourself +for safety: + +```ts +interface KpmDetect { pose: number[]; page: number; error: number } +interface NftTrack { found: boolean; matrix: number[]; error: number; cont_num: number } + +const d = kpm.detect(rgba) as KpmDetect | null; +const t = nft.track(rgba, width, height) as NftTrack; +``` + +## Memory management + +Each handle owns WebAssembly-side memory. wasm-bindgen generates a `free()` +method on every exported class — call it when you are done with a handle +(e.g. when unmounting a component or switching markers) so the memory is +released deterministically instead of waiting for GC: + +```js +kpm.free(); +nft.free(); +``` + +Internal C-side resources (camera-parameter lookup tables, ICP handles) are +released by the Rust `Drop` implementations when the handle is freed. + +## Common pitfalls + +- **RGBA length must match exactly.** `detect()` and `track()` expect + `width * height * 4` bytes and reject anything else. Feed canvas + `ImageData.data` at the same dimensions you constructed the handle with. +- **Load before detect/track.** `detect()` errors if `load_ref_data()` hasn't + run; `track()` errors if `load_nft_marker()` hasn't run. Check + `is_loaded()` if unsure. +- **Single marker per handle.** `load_ref_data()` remaps all pages to page 0, + so one `WasmKpmHandle` tracks one marker. Use multiple handles for multiple + markers. +- **`track()` needs a prior pose.** Call `set_initial_pose()` (from KPM, or a + previous frame) before the first `track()`; without an initial pose it + returns `{ found: false }`. +- **Frame size is fixed per handle.** The constructor scales camera params to + the size you pass. If your video resolution changes, build a new handle. + +## See also + +- [`simple_nft`](../crates/core/examples/simple_nft.rs) — the native + equivalent of this detect-then-track flow. +- The `crates/wasm/www/` demos (`simple.html`, + `simple_nft_example.html`, `simple_video_marker_example.html`) — runnable + browser examples. +- [jsartoolkitNFT](https://github.com/webarkit/jsartoolkitNFT) — the + Emscripten/C++ stack this pure-Rust port matches for pose output.