From d5188959a21224151e54e336930ccfba5c3c9620 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 31 Aug 2026 16:15:25 +0000 Subject: [PATCH] docs: add Character Kits operator guide and API contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the workspace Character Kit library, Face Rig overlays, cutout dialogue cadence, and recipe inventory. Cross-link 3D Video / README and record Director pipeline pagination plus the stale Scene→Recipe audit. Co-authored-by: ignaciodelcano+dcl --- README.md | 8 +- app/docs/API.md | 34 +++- docs/3d-video-compositor/HOWUSEIT.md | 13 +- docs/HOWUSEIT.md | 1 + docs/character-kits/HOWUSEIT.md | 247 +++++++++++++++++++++++++++ docs/scene-recipe-gap.md | 12 +- docs/workspaces/HOWUSEIT.md | 5 +- 7 files changed, 310 insertions(+), 10 deletions(-) create mode 100644 docs/character-kits/HOWUSEIT.md diff --git a/README.md b/README.md index 2cb0810f..e9a389f7 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,9 @@ HocusPocus exists for creators who do not want a one-off prompt machine. It is a 2. **Director (sidebar)** — select Music Video, Short Film, Trailer or a story-driven workflow and describe the outcome. The LLM turns the brief into reviewable shots, prompts and references. Choose manual review for control or automatic mode for a complete recoverable pipeline. 3. **Gallery: All, Images, Videos, Audio, Videoclips, Trailers and Chapters** — browse results by kind. Open an item to inspect it; use it as a reference, send it to an editor, or keep it in the active workspace. 4. **3D** — create a mesh from text, one image or the four front/left/right/back views. You can upload references or select existing HocusPocus images. Export GLB for later animation or 3D-video composition. -5. **3D Video** — place GLBs, images and effects in a controllable scene and render an MP4/WebM composition. Use it for camera moves that must be exact rather than invented by a video model. +5. **3D Video** — place GLBs, images, Character Kits and effects in a controllable scene and render an MP4/WebM composition. Use it for camera moves that must be exact rather than invented by a video model, and for 2D cutout speech (Face Rig + dialogue beats). 6. **Animate** — rig a compatible static GLB and apply procedural or AI-assisted motion. Export the rigged model or bring it back to 3D Video. -7. **Character Creator** — upload one subject image. H3 makes a centered 360° turntable; select/re-take the best front, left, back and right frames, then build a Hunyuan multi-view mesh. It is the quickest route from character art to a 3D asset. +7. **Character Creator** — upload one subject image. H3 makes a centered 360° turntable; select/re-take the best front, left, back and right frames, then build a Hunyuan multi-view mesh. **Create / open CharacterKit Face Rig** hands a saved view to **3D Video** as a 2D puppet (not a mesh). Operator guide: [Character Kits / Face Rig](docs/character-kits/HOWUSEIT.md). 8. **Story Lab** — write the production bible: premise, world rules, locations, cast, relationships and beats. Review each field, then hand the approved canon to Comic Studio, Director, trailers or music-video production. 9. **Series Lab** — turn approved canon into seasons, episodes, scenes and shots. Use it when continuity has to survive several episodes and you need attempts and references tracked per shot. 10. **Workspaces** — separate client work, worlds and experiments. The workspace selector changes the active output folder; this tab is also the place to inspect and resume Director production threads. @@ -200,7 +200,7 @@ Included geometry variants: - **Hunyuan3D 2 Multi-view**: Turbo, Fast, and full-step models using front/left/right/back references - **Hunyuan3D 2.1**: high-fidelity 3.3B geometry with optional PBR materials -The **Characters** tab builds a Hunyuan multi-view mesh from one subject or object photo: MiniMax-M3 vision writes the A Prompt (`POST /api/v1/characters/describe-refs`), H3 Ref2VA records a frozen 360° orbit (`768x1344`, 124 frames), then four stills at frames 2 / 21 / 42 / 63 become front / left / back / right for `hunyuan3d-2mv-turbo`. Details: **[3D Video compositor](docs/3d-video-compositor/HOWUSEIT.md)** §5.4. +The **Characters** tab builds a Hunyuan multi-view mesh from one subject or object photo: MiniMax-M3 vision writes the A Prompt (`POST /api/v1/characters/describe-refs`), H3 Ref2VA records a frozen 360° orbit (`768x1344`, 124 frames), then four stills at frames 2 / 21 / 42 / 63 become front / left / back / right for `hunyuan3d-2mv-turbo`. Details: **[3D Video compositor](docs/3d-video-compositor/HOWUSEIT.md)** §5.4. The same tab can hand a saved view to **3D Video** as a Character Kit Face Rig (2D cutout, not a mesh): **[Character Kits](docs/character-kits/HOWUSEIT.md)**. The advanced panel exposes inference steps, guidance, octree resolution, processing chunks, seed, texture model/resolution, CPU offload, FlashVDM, Torch compilation, DMC/Marching Cubes, mesh simplification, face target, and GLB/OBJ/PLY/STL export. Four presets provide sensible Low VRAM, Balanced, Quality/PBR, and Multi-view configurations. @@ -439,7 +439,7 @@ After clicking **Start**, the launcher shows an **Open Web UI** button once the - **Activity footer** — persistent live job progress and access to current or past **Productions** - **Settings drawer** (gear icon) — model visibility, performance auto-tune, services (LLM, API keys, NSFW, theme) - **Pinokio menu** — Update, Reset, Install Inpaint Support, LoRA folder shortcuts -- **Operator guides** — [HOWUSEIT index](docs/HOWUSEIT.md) (Video Editor, Workspaces tab, 3D compositor) +- **Operator guides** — [HOWUSEIT index](docs/HOWUSEIT.md) (Video Editor, Workspaces tab, 3D compositor, Character Kits) ## Sharing on the local network diff --git a/app/docs/API.md b/app/docs/API.md index b191d2ab..e10be268 100644 --- a/app/docs/API.md +++ b/app/docs/API.md @@ -717,13 +717,43 @@ curl -X POST "$MAESTRO_URL/api/v1/characters/describe-refs" \ `kind` is `character` or `object` (default `character`). `roles` are `subject`, `face`, `outfit`, `extra`, or `accessory`; missing/invalid roles become `subject` for index 0 and `extra` otherwise. Response: `{ "a_prompt", "kind" }`. `400` if `image_paths` is empty, a file is missing, or the API key is unset. +## Character Kits + +Reusable 2D cutout puppets live in `{workspace}/.character-kit-library-v1.json` (max 100 kits, 20 MB). Writes are compare-and-swap on `revision`. Operator guide: `docs/character-kits/HOWUSEIT.md`. + +- `GET /api/v1/character-kits/library?workspace=default` — load or return an empty `{ version: 1, revision: 0, activeId: "", kits: {} }`. +- `PATCH /api/v1/character-kits/library/kits/{kit_id}` — `{ workspace, baseRevision, kit, makeActive? }`. `makeActive` defaults true. `400` invalid kit. `409` `{ "code": "character_kit_revision_conflict", "expectedRevision", "currentRevision" }`. +- `DELETE /api/v1/character-kits/library/kits/{kit_id}` — `{ workspace, baseRevision }`. `404` if missing. Source files are not deleted. +- `POST /api/v1/character-kits/face-rig/cleanup` — `{ workspace, source, padding? }` (`padding` 0–64, default 8). rembg U2Net + crop-to-alpha; writes a new PNG and never overwrites `source`. `400` / `404` if the path is outside uploads/workspace. + +Workspace token: `default` or `[A-Za-z0-9][A-Za-z0-9_-]*`. Mouth keys are only `closed|small|wide|round`. `blob:` sources are rejected. `lookNotes` is UI-only and is stripped on save. + +```bash +curl -X PATCH "$MAESTRO_URL/api/v1/character-kits/library/kits/luma" \ + -H "Content-Type: application/json" \ + -d '{ + "workspace": "default", + "baseRevision": 0, + "kit": { + "version": 1, "id": "luma", "name": "Luma", "style": "cutout", + "base": { + "id": "luma-base", "name": "Luma base", "source": "luma-base.png", + "kind": "image", "alphaStatus": "transparent", "reviewState": "approved" + }, + "poses": {}, "mouth": {}, "eyes": {}, + "anchors": { "base": { "mouth": { "offsetX": 0, "offsetY": -18, "scale": 0.05, "rotation": 0 } } }, + "provenance": [] + } + }' +``` + ## Director pipeline threads These routes always use the server active workspace. They do not accept `?workspace=`. -- `GET /api/v1/director/pipelines` / `GET /api/v1/director/pipelines/active` / `GET /api/v1/director/pipelines/{pid}` — list or load. `{pid}` hydrates an empty `clips` array from `clip_plans` or `planned_clips` and sets `queue_source` to `clips`, `clip_plans`, or `planned`. +- `GET /api/v1/director/pipelines` / `GET /api/v1/director/pipelines/active` / `GET /api/v1/director/pipelines/{pid}` — list or load. `{pid}` hydrates an empty `clips` array from `clip_plans` or `planned_clips` and sets `queue_source` to `clips`, `clip_plans`, or `planned`. List accepts `limit` and `offset` (newest first). `limit=0` (the default) returns the full list; the Workspaces tab pages 8 at a time and reads `total` for “load more”. - `PUT /api/v1/director/pipelines/{pid}/clips/{clip_index}/prompt` — optional `video_prompt`, `image_prompt`, `soundtrack_drive`. `true` writes an `audio_driven` / `lip_sync_critical` plan; `false` writes `music_driven` and clears `_director_dialogue_beats`. `409` while the pipeline is active. - `POST /api/v1/director/pipeline/{pid}/resume` and `POST /api/v1/director/pipeline/{pid}/continue` use the singular `pipeline` path. - Batch prompt rewrite is UI-only: loop `POST /api/v1/llm/generate` (local LLM) then PUT the chosen prompts. -Operator notes: `docs/video-editor/HOWUSEIT.md` and `docs/workspaces/HOWUSEIT.md`. +Operator notes: `docs/video-editor/HOWUSEIT.md`, `docs/workspaces/HOWUSEIT.md`, and `docs/character-kits/HOWUSEIT.md`. diff --git a/docs/3d-video-compositor/HOWUSEIT.md b/docs/3d-video-compositor/HOWUSEIT.md index 7f642a7e..d9c8f583 100644 --- a/docs/3d-video-compositor/HOWUSEIT.md +++ b/docs/3d-video-compositor/HOWUSEIT.md @@ -19,8 +19,9 @@ A **layered compositor**, not MiniMax H3. | Scene Animator | **3D Video** | Stack images, videos, GLBs, camera, rain/fog/etc. Animate them. Record a clip | | MiniMax H3 | Studio / Story Lab | Native video + stereo audio (acting, dialogue, locations) | | Video Editor | **Video Editor** | Join compositor clips with H3 clips | +| Character Kits | **3D Video** sidebar | Reusable 2D cutout puppets + Face Rig mouths. Operator guide: [Character Kits](../character-kits/HOWUSEIT.md) | -Use the compositor when you need **controllable motion of a known object** over plates: a ship crossing stars, a UFO rising behind mountains, a logo flying in, rain over a still. Use H3 when you need **performance, speech, or a living location**. Mix them: H3 for people/places, compositor for the vehicle insert, Video Editor to cut them together. +Use the compositor when you need **controllable motion of a known object** over plates: a ship crossing stars, a UFO rising behind mountains, a logo flying in, rain over a still. Use H3 when you need **performance, speech, or a living location**. Mix them: H3 for people/places, compositor for the vehicle insert, Video Editor to cut them together. Use a **Character Kit** when the “known object” is a graphic puppet that must speak with four mouth sprites — not a Hunyuan mesh and not H3 lipsync. Do **not** ask H3 to “keep this exact GLB flying on a perfect path.” H3 will invent a new ship. The compositor keeps the mesh. @@ -232,6 +233,14 @@ Import H3 MP4s + recorded WebM. Multi-select in **From Loreframe Lab**. Export M Director / Series **auto-joins** (not the editor) use a 0.5 s last-frame freeze + ~0.4 s crossfade when there is no driving audio. Full probe/export/`result_kind` contract: [Video Editor / mixes](../video-editor/HOWUSEIT.md). +### 5.9 Character Kits (2D puppets) + +Workspace library: `GET/PATCH/DELETE /api/v1/character-kits/library…`. Overlay matte: `POST /api/v1/character-kits/face-rig/cleanup`. Character Creator can hand a saved view into this editor (`hocuspocus:character-kit-face-rig-handoff`) but does not generate visemes itself. + +Only **approved** poses/mouths mount into the scene or appear in the Recipe inventory (`APPROVED_CHARACTER_KIT`). Spoken cutout dialogue is `scene.dialogueBeats` compiled to hold/snap opacity keyframes — not phoneme-perfect lipsync. + +Full contract, CAS revisions, mouth packs, and pitfalls: [Character Kits / Face Rig](../character-kits/HOWUSEIT.md). + --- ## 6. Layer cookbook (what to stack) @@ -421,6 +430,7 @@ Do this **in the browser tab**, not as a Python overnight script. UI: `SceneReci 4. **Auto**: **Generate + compose** creates missing plates/meshes. One `identity` per object — a UFO series uses **one** GLB and several `shots[]`. Static environments use image plates; inherently moving scenery can use an H3 video plate. Rain, fog, snow and particles use procedural effects instead of redundant generated overlays. Default `record`/`save` are false so you preview first. 5. GPU jobs and Hunyuan poll with timeouts; **Cancel** aborts the run. If Lab dies (segfault), the runner errors instead of spinning forever. 6. After compose, switch shots in the recipe panel without regenerating the mesh. A recipe rig `clip` is mounted into the Scene Animator and disables unintended turntable spin. +7. Approved Character Kits enter inventory as `APPROVED_CHARACTER_KIT` rows. Spoken cutout shots need both a `speech` audio entry and a `dialogueBeats` row whose `mouthLayerIds` name the overlays (for `cutout-talking-head`: `["mouth-open", "mouth-closed"]`). Never mix one kit’s mouth with another kit’s body. Keep the 3D Video tab visible while it records. WebM is the recorded format; import it in Video Editor to join with H3 MP4s. @@ -440,6 +450,7 @@ Keep the 3D Video tab visible while it records. WebM is the recorded format; imp | `app/services/rig_service.py` | Rig profiles and clips | | `docs/minimax-h3-prompting.md` | H3 prompt dialect | | `docs/video-editor/HOWUSEIT.md` | Cut compositor WebM with H3 MP4s; mix kinds | +| `docs/character-kits/HOWUSEIT.md` | Character Kit library, Face Rig, cutout dialogue | | `ui/src/features/characters/orbitPrompt.ts` | Orbit A/B prompts and still-frame indices | When in doubt: **one identity per mesh, one path per compositor shot, H3 never draws that mesh.** diff --git a/docs/HOWUSEIT.md b/docs/HOWUSEIT.md index b5bf8f4e..90d29f80 100644 --- a/docs/HOWUSEIT.md +++ b/docs/HOWUSEIT.md @@ -5,3 +5,4 @@ Operator guides for Loreframe Lab subsystems. Prefer these over inventing a seco - **Video Editor** (import Lab clips, timeline, export) and **assembled mixes** (`result_kind` gallery tabs): [`docs/video-editor/HOWUSEIT.md`](video-editor/HOWUSEIT.md) - **Workspaces tab** (Director generation threads — not the output-directory switcher): [`docs/workspaces/HOWUSEIT.md`](workspaces/HOWUSEIT.md) - **3D Video compositor** (Hunyuan meshes + plates + camera + rain/fog, mixed with MiniMax H3): [`docs/3d-video-compositor/HOWUSEIT.md`](3d-video-compositor/HOWUSEIT.md) +- **Character Kits / Face Rig** (workspace 2D puppets, viseme overlays, cutout dialogue): [`docs/character-kits/HOWUSEIT.md`](character-kits/HOWUSEIT.md) diff --git a/docs/character-kits/HOWUSEIT.md b/docs/character-kits/HOWUSEIT.md new file mode 100644 index 00000000..9474f70f --- /dev/null +++ b/docs/character-kits/HOWUSEIT.md @@ -0,0 +1,247 @@ +# HOWUSEIT — Character Kits and Face Rig + +Operator guide for reusable **2D cutout puppets**: one reviewed body/pose, mouth visemes, an optional blink, and pose-local face anchors. This is not Character Creator’s Hunyuan turntable and not MiniMax H3 lip-sync. + +UI: **3D Video** sidebar (`SceneAnimatorPanel` → Character Kits). Code: `ui/src/lib/characterKit.ts`, `ui/src/lib/characterKitFaceRig.ts`, `ui/src/lib/cutoutDialogue.ts`. Persistence: `app/services/character_kit_library.py`. Cleanup: `POST /api/v1/character-kits/face-rig/cleanup`. HTTP: `app/_launch_runtime.py`. + +Related: [3D Video compositor](../3d-video-compositor/HOWUSEIT.md), [Character Creator orbit](../3d-video-compositor/HOWUSEIT.md) §5.4. + +--- + +## 1. What this system is + +| Tool | Tab | Job | +|---|---|---| +| Character Creator | **Characters** | Identity photo → H3 360° orbit → Hunyuan multi-view **mesh** | +| Character Kit library | **3D Video** | Workspace-scoped 2D puppet (base pose + mouths + blink) | +| Face Rig | **3D Video** → Paso 2 · Labios / ojos | Generate, clean, place, and review overlays | +| Cutout dialogue | **3D Video** | Held/snap mouth keyframes from known text or speech | +| Recipe runner | **3D Video** | Compiles `dialogueBeats` and only **approved** kit pieces | + +Use a kit when you need a **repeatable graphic character** that can speak with four mouth sprites. Use Character Creator when you need a **3D mesh**. Use H3 when you need a **performed** face, not a paper flap. + +The compositor never claims phoneme-perfect lipsync. Cadence is bounded and graphic. + +--- + +## 2. Hard limits + +1. **Library file** is `{workspace}/.character-kit-library-v1.json`. Max **100** kits, **32** poses, **20 MB** encoded JSON, **500** provenance objects per kit. +2. **Compare-and-swap.** Every write sends `baseRevision`. Stale clients get `409` `{ "code": "character_kit_revision_conflict", "expectedRevision", "currentRevision" }`. Reload the library; do not retry the same revision. +3. **No blob/data URLs.** Sources must be persistent (`/api/v1/file/…`, `/api/v1/uploads/…`, or a workspace filename). Face Rig handoff from Character Creator also rejects transient images. +4. **Only `approved` pieces mount or enter the recipe inventory.** Generated Face Rig states start as `pending`. Cleanup does not approve. Mouth-pack apply does not approve. +5. **Mouth states** are only `closed` | `small` | `wide` | `round`. Eyes are `open` | `blink` (`open` is unused by the current compiler). Unknown keys `400`. +6. **Anchors are pose-local**, not 16:9 frame percents. Default mouth `{ offsetX: 0, offsetY: -18, scale: 0.05 }`. Default blink `{ offsetX: 0, offsetY: -28, scale: 0.12 }`. Bounds: offset ±200, scale 0.001–20, rotation ±360. +7. **Workspace token** is `default` or `[A-Za-z0-9][A-Za-z0-9_-]*`. Kits follow the output-directory workspace, not the Workspaces *tab*. +8. **`lookNotes` is not persisted.** The UI keeps style/traits for the current editor session. `normalize_character_kit` drops unknown fields, including `lookNotes`. After reload, re-enter style + traits before regenerating overlays. +9. **Deleting a kit does not delete files.** Pose PNGs, cleaned overlays, and scene layers stay on disk. + +--- + +## 3. Data model + +``` +CharacterKitLibrary { version: 1, revision, activeId, kits{} } + +CharacterKit + id, name, style: cutout | children-illustration | anime-2d + identityReference?, base?, poses{} + mouth { closed?, small?, wide?, round? } + eyes { blink? } + anchors { [poseId]: { mouth, mouthStates?, eyes? } } + provenance[] +``` + +Each asset: `{ id, name, source, kind: image|overlay, alphaStatus, reviewState, prompt?, model?, workspace? }`. + +`alphaStatus`: `unknown` | `transparent` | `opaque`. Transparent if ≥1% of pixels have alpha < 250. + +Mount (`mountCharacterKitLayers`) parents each approved overlay to the pose, sets `faceBinding`, and starts with closed mouth visible. Re-mounting the same kit pose ids into a scene that already has them fails. + +Recipe inventory flattens only approved pieces. The **active** kit is ordered first so a large workspace does not evict its complete face set under the global inventory cap. + +--- + +## 4. Operator workflow + +### 4.1 From Character Creator + +1. Stay on **character** (not object). Capture a turnaround view or upload the subject. +2. **Create / open CharacterKit Face Rig** writes `sessionStorage` key `hocuspocus:character-kit-face-rig-handoff` and switches to **3D Video**. +3. If a kit already uses that source as `base` or `identityReference`, the editor reopens it. Otherwise it drafts a new kit with the pose **already approved**. + +Character Creator itself does not generate visemes. Error if kind is `object`: *Face Rig is for Character Kits.* + +### 4.2 From a compositor layer + +1. Open **3D Video**. Add a full-body cutout (generated image or transparent PNG). +2. **New kit from selected base layer**, or assign **Selected → base / pose / mouth / blink**. +3. Review alpha (transparent vs opaque). Approve the pose before Face Rig generation. +4. **Paso 2 · Labios / ojos**: style chips + traits → generate pose (if needed) or generate each viseme. Optional: apply a mouth pack, **Clean**, place, **Lock all mouths**, then approve. +5. **Save kit** (PATCH). **Mount pose** into the current scene. + +### 4.3 Face Rig generation + +Prompts are built in `characterKitPosePrompt` / `faceRigPrompt`. The user only supplies style and traits; the helper asks for an isolated transparent overlay (or a full-body standing cutout for the pose). + +Image jobs use the Studio image model with `strictReference: true` and the approved pose as identity. Negative prompt forbids a full head/body/skin rectangle. + +**Mouth packs** (no GPU): `GET /character-kit-presets/mouths/manifest.json`. Packs: `paper-cut`, `children-illustration`, `limited-anime`, `felt-puppet`, `comic-ink`, `watercolor`. Apply attaches pending overlays; you still place and approve them. + +**Wipe mouth box** paints an ellipse with nearby skin samples and registers a new pose file (`character-kit-mouth-wipe`). It does not delete the original. + +**Cleanup** (`POST /api/v1/character-kits/face-rig/cleanup`): rembg U2Net + crop-to-alpha. `padding` 0–64 (default 8). Writes `{stem}.cleanup-{8hex}.png` and never overwrites the original. `400` if the source is outside uploads/workspace or the matte is empty. + +Placement warnings (`assessFaceRigPlacement`) never auto-approve. Typical mouth scale is ≤ 0.12; blink ≤ 0.20. Mouths on a full-body cutout usually sit above the chest (`offsetY` more negative than −8). + +**Lock all mouths** copies one calibrated mouth box onto `closed/small/wide/round` for that pose. + +### 4.4 Preview speech (Face Rig only) + +`previewFaceRigDialogue` plans **2–4 s** of visemes with the same cadence as scene dialogue. Missing shapes fall back (`wide` → `small` → `round` → `closed`). This preview does **not** write scene keyframes. + +--- + +## 5. Cutout dialogue (scene) + +Requires at least one speaking overlay (`Open` / `Small` / `Wide` / `Round`). Optional `Closed`. Bind overlays to the selected pose first (`faceBinding` + `parent`). + +| Action | Result | +|---|---| +| Animate from line | `planCutoutDialogue(text, start, end, fps)` → opacity keyframes | +| Detect from audio | speech units → per-word plans, then the same compiler | + +Cadence (verified in `cutoutDialogue.ts`): + +- Minimum hold `max(2/fps, 0.12s)`. +- Viseme from glyphs: consonants/punctuation → `closed`; `o/u` → `round`; `a/e` → `wide`; other vowels → `small`. +- First and last beats are **closed** so cuts do not freeze on an open mouth. +- Very short words with a vowel get one centre pulse so they are not closed/closed after the edge guard. +- Missing viseme sprites fall back to Open / Wide / Small / Round, in that order. + +Persisted on the scene as `dialogueBeats[]`: + +```json +{ + "id": "beat-1", + "text": "The square is frozen.", + "start": 0.4, + "end": 2.8, + "mouthLayerIds": ["kit-luma-mouth-wide", "kit-luma-mouth-closed"], + "audioTrackId": "speech-1", + "confidence": "known-text" +} +``` + +`confidence` is `known-text` | `aligned-audio` | `energy-fallback`. Editing text, speaker, or timing **recompiles** mouth keyframes (`rebuildCutoutDialogueLayers`). Never target the hero/base plate as a mouth layer. + +Narrative templates that already ship mouth slots: `cutout-talking-head` (`mouth-open` / `mouth-closed`) and `cutout-speaking-blink`. Hold/snap-only templates (`cutout-dialogue-hold`, `cutout-reaction-snap`) do **not** flap a mouth. + +--- + +## 6. Recipes + +The Recipe runner (`ui/src/lib/sceneRecipe.ts`) receives approved kit inventory tagged `APPROVED_CHARACTER_KIT id=…; role=base|pose/…|mouth/…|eyes/…`. + +Rules the LLM is given: + +- Keep body and face pieces from the **same kit id**. +- Spoken cutout dialogue needs a `speech` audio entry **and** a top-level `dialogueBeats` entry with `audioTrackId`, exact text, time range, and `mouthLayerIds`. +- Multi-shot recipes must set per-shot `audioTrackIds` and `dialogueBeatIds`. `[]` means silent / no mouths. Do not copy the full mix into every shot. + +Compiler: `compileRecipeDialogue` → ordinary hold keyframes. Export sidecar stores the compiled scene plus the recipe. + +--- + +## 7. HTTP + +Workspace query on GET; body field on writes. Routes do **not** use the Director “active workspace only” rule — pass the output-directory name. + +### `GET /api/v1/character-kits/library?workspace=default` + +Returns the normalized library, or an empty `{ version: 1, revision: 0, activeId: "", kits: {} }` when the file is missing. + +```bash +curl "$MAESTRO_URL/api/v1/character-kits/library?workspace=default" +``` + +### `PATCH /api/v1/character-kits/library/kits/{kit_id}` + +Create or replace **one** kit. Neighbours are untouched. `makeActive` defaults true. + +```bash +curl -X PATCH "$MAESTRO_URL/api/v1/character-kits/library/kits/luma" \ + -H "Content-Type: application/json" \ + -d '{ + "workspace": "default", + "baseRevision": 0, + "makeActive": true, + "kit": { + "version": 1, + "id": "luma", + "name": "Luma", + "style": "cutout", + "base": { + "id": "luma-base", + "name": "Luma base", + "source": "luma-base.png", + "kind": "image", + "alphaStatus": "transparent", + "reviewState": "approved" + }, + "poses": {}, + "mouth": {}, + "eyes": {}, + "anchors": { + "base": { "mouth": { "offsetX": 0, "offsetY": -18, "scale": 0.05, "rotation": 0 } } + }, + "provenance": [] + } + }' +``` + +`400` invalid body. `409` revision conflict. + +### `DELETE /api/v1/character-kits/library/kits/{kit_id}` + +Same CAS. Body: `{ "workspace", "baseRevision" }`. `404` if the id is missing. + +### `POST /api/v1/character-kits/face-rig/cleanup` + +```bash +curl -X POST "$MAESTRO_URL/api/v1/character-kits/face-rig/cleanup" \ + -H "Content-Type: application/json" \ + -d '{ "workspace": "default", "source": "mouth-wide.png", "padding": 8 }' +``` + +Response includes `filename`, `source`, `original`, `width`, `height`, `alpha`, `method: "rembg-u2net"`, `padding`. `400` / `404` if the file is not permitted. + +--- + +## 8. Pitfalls + +- Treating Face Rig as a 3D blendshape or as H3 speech. It is four PNG overlays + opacity holds. +- Mounting before review → `Review and approve … before mounting`. +- Two browser tabs saving the same workspace → `409`; the second tab must reload. +- Mixing kit A’s mouth with kit B’s body in a recipe. The inventory text forbids it; the compiler will not catch a hand-edited JSON mix. +- Naming overlays without `mouth` / viseme tokens and without `faceBinding`. Discovery falls back to labels; kit mounts set `faceBinding` explicitly. +- Expecting `lookNotes` to survive Save kit. +- Running cleanup on a pose (full body). The endpoint mattes **one overlay**; it will crop a whole character to its opaque bbox. +- Character Creator object mode → Face Rig. Rejected on purpose. + +--- + +## 9. Files to read next + +| Path | Why | +|---|---| +| `ui/src/lib/characterKit.ts` | Types, mount, recipe inventory | +| `ui/src/lib/characterKitFaceRig.ts` | Prompts, packs, anchors, wipe, preview | +| `ui/src/lib/cutoutDialogue.ts` | Viseme planner + keyframe compiler | +| `ui/src/lib/characterKitHandoff.ts` | Creator → 3D Video session handoff | +| `ui/src/lib/sceneRecipe.ts` | `dialogueBeats` + `APPROVED_CHARACTER_KIT` | +| `app/services/character_kit_library.py` | CAS store and validation | +| `app/services/character_kit_face_cleanup.py` | rembg + crop | +| `ui/public/character-kit-presets/mouths/manifest.json` | Pack ids and files | +| `tests/test_character_kit_library.py` | Server contract | +| `ui/tests/characterKitFaceRig.test.mjs` | Client Face Rig contract | diff --git a/docs/scene-recipe-gap.md b/docs/scene-recipe-gap.md index 541f73d6..d94cb31b 100644 --- a/docs/scene-recipe-gap.md +++ b/docs/scene-recipe-gap.md @@ -4,12 +4,20 @@ Field-by-field audit of everything a `Scene` can hold against everything a `SceneRecipe` can express. Six readers swept one domain each, and every claim was then re-checked by a second reader briefed to refute it. +**Status (2026-08-31):** a Scene → Recipe serializer now exists at +`ui/src/lib/sceneToRecipe.ts`. It copies `animation.keyframes`, `strip` / +`seamOccluder`, `relationship`, `visible`, `locked`, `faceBinding`, per-layer +`effects`, and related timing. The counts below are the **T2.3 snapshot** from +before that serializer; do not treat “there is no serializer” as current. +Re-audit before using this file as a blocker list. Character Kit / cutout +dialogue transport is documented in [`character-kits/HOWUSEIT.md`](character-kits/HOWUSEIT.md). + **122 fields audited. 65 cannot be expressed at all, 28 partially, 29 fully.** 118 claims survived verification; 4 were corrected. -There is no `Scene → Recipe` serializer anywhere in `ui/src`. The sidecar written +At audit time there was no `Scene → Recipe` serializer in `ui/src`. The sidecar written beside every exported MP4 pairs the recipe with the compiled scene and presents -the recipe as the clip's reproduction. For 65 fields that is already untrue, and +the recipe as the clip's reproduction. For 65 fields that was already untrue, and it becomes untrue for a given clip the moment anyone edits it. ## Where the losses cluster diff --git a/docs/workspaces/HOWUSEIT.md b/docs/workspaces/HOWUSEIT.md index e19cec1b..f4df9326 100644 --- a/docs/workspaces/HOWUSEIT.md +++ b/docs/workspaces/HOWUSEIT.md @@ -23,10 +23,12 @@ Typical flow: generate a song or Director video elsewhere → the thread appears ## 2. Queue inspection -`GET /api/v1/director/pipelines` — saved threads. +`GET /api/v1/director/pipelines` — saved threads (newest first). `GET /api/v1/director/pipelines/active` — in-memory runs (recovery). `GET /api/v1/director/pipelines/{pid}` — full state, **hydrated**. +List query: `?limit=&offset=`. **`limit=0` (default) returns every saved pipeline** and parses each JSON. The Workspaces UI pages **8** (`DASHBOARD_PIPELINE_PAGE_SIZE`) and uses `total` + `loadMorePipelineList` so opening the tab does not hydrate the whole archive. `GET …/{pid}` is still required for the selected thread. Status polls are serialised in the UI; do not fire a full unpaged list on an interval. + Hydration (`hydrate_queue_clips`): | Condition | `queue_source` | @@ -128,6 +130,7 @@ Rejoin uses the mix path in [Video Editor / mixes](../video-editor/HOWUSEIT.md) ## 7. Pitfalls +- Calling `GET /api/v1/director/pipelines` with the default `limit=0` from a poller. That re-parses every pipeline JSON. Use `limit`/`offset` (the tab uses 8). - Confusing this tab with the **output-directory** switcher. Threads are scoped to whichever directory is active. - Editing prompts while Director is sampling → `409`. - Batch rewrite without a loaded local LLM → `POST /api/v1/llm/generate` fails.