Add F06_MUSE spec: in-browser avatar illustrator - #32
Merged
Conversation
Design-only spec for a client-side rebuild of Draft's rotating-avatar feature: photo upload, MediaPipe reuse for face detection, and a runtime-fetched (not vendored) SD-Turbo diffusion model for stylization and reposing. Argues the site's first same-origin exception for an ML dependency, since the model is far too large to vendor under existing precedent. Flags img2img feasibility, real download size, and real- hardware latency as unverified premises pending a working prototype.
Prototyped the vae_encoder path in a real browser session, resolving F06-AC01 from unverified to confirmed. Surfaces two new findings: the encoder's inputs/outputs are float16-only (undocumented), and this build's WebGPU EP has no fp16 Clip kernel for it, forcing a ~52s wasm fallback that is now the dominant latency cost for AC03.
Re-ran the img2img prototype against onnxruntime-web 1.20.1 instead of the reference demo's stale 1.18.0-dev pin. PR #21584 (merged 2024-08-28, shipped in 1.19.2+) added the missing fp16 WebGPU Clip kernel: vae_encoder now runs on WebGPU in 595ms with no CPU fallback, and the full pipeline completes in ~2.5s. Only remaining premise is confirming that number on real consumer hardware.
The prototype's browser automation runs on this machine, not an isolated sandbox. Confirmed hardware via system_profiler and the WebGPU adapter info, then re-ran the full pipeline twice: ~2-2.5s total end-to-end, all four sessions on WebGPU. Noted as a best-case number since an M4 Pro is a premium device, not the low end of "ordinary hardware."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
specs/PRD.md's feature list, status "proposed, not implemented".No code changes — this is a design document only, following the
F0N_NAME.mdtemplateF05_IRIS.mdestablished.Test plan
.htmlpage was added yet.