Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ HocusPocus exists for creators who do not want a one-off prompt machine. It is a

## First steps: guide to every section

1. **Studio (sidebar)** — choose an image, video or audio model; write the prompt; add references/LoRAs; then generate. Use it when you want direct, manual control over one asset. The output appears in the gallery and is reusable everywhere else.
1. **Studio (sidebar)** — choose an image, video or audio model; write the prompt; add references/LoRAs; then generate. Use it when you want direct, manual control over one asset. The output appears in the gallery and is reusable everywhere else. **Tools** (same sidebar) post-processes an existing image or clip — FlashVSR/Lanczos upscale, SeedVC revoice, rembg background removal — and always writes a new file; see the [Studio Tools guide](docs/tools/HOWUSEIT.md).
2. **Director (sidebar)** — select Music Video, Short Film, Trailer or a story-driven workflow and describe the outcome. The LLM turns the brief into reviewable shots, prompts and references. Choose manual review for control or automatic mode for a complete recoverable pipeline.
3. **Gallery: All, Images, Videos, Audio, Videoclips, Trailers and Chapters** — browse results by kind. Open an item to inspect it; use it as a reference, send it to an editor, or keep it in the active output folder.
4. **3D** — create a mesh from text, one image or the four front/left/right/back views. You can upload references or select existing HocusPocus images. Export GLB for later animation or 3D-video composition.
Expand Down Expand Up @@ -439,7 +439,7 @@ After clicking **Start**, the launcher shows an **Open Web UI** button once the
- **Activity footer** — persistent live job progress and access to current or past **Productions**
- **Settings drawer** (gear icon) — model visibility, performance auto-tune, services (LLM, API keys, NSFW, theme)
- **Pinokio menu** — Update, Reset, Install Inpaint Support, LoRA folder shortcuts
- **Operator guides** — [HOWUSEIT index](docs/HOWUSEIT.md) (Video Editor, Workspaces tab, 3D compositor, Character Kits)
- **Operator guides** — [HOWUSEIT index](docs/HOWUSEIT.md) (Studio Tools, Video Editor, Workspaces tab, 3D compositor, Character Kits)

## Sharing on the local network

Expand Down
56 changes: 44 additions & 12 deletions app/docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -699,23 +699,26 @@ status = requests.get(f"{base}/api/v1/video-editor/export/{job['job_id']}").json

## Image background removal

Operator workflow: [`docs/tools/HOWUSEIT.md`](../../docs/tools/HOWUSEIT.md).

`POST /api/v1/tools/remove-background` queues a standalone image tool job. It
uses the shared rembg U2Net adapter, never overwrites the source, and publishes
the transparent PNG plus a canonical `.meta.json` asset manifest in the
destination workspace. Use an exact `asset_id` from `GET /api/v1/assets?kind=image`
whenever possible; `source` may be the exact filename, an `/api/v1/file/...`
URL, or an absolute path already inside the selected uploads/workspace root.
`source_workspace` is required when the source belongs to another output
folder. Poll `GET /api/v1/status/{job_id}` and cancel with
`POST /api/v1/cancel/{job_id}`.
destination workspace. Accepted image extensions are `.png`, `.jpg`, `.jpeg`,
and `.webp` (narrower than Tools upscale). Use an exact `asset_id` from
`GET /api/v1/assets?kind=image` whenever possible; `source` may be the exact
filename, an `/api/v1/file/...` URL, or an absolute path already inside the
selected uploads/workspace root. For a source in another output folder,
preserve its file-URL `?workspace=` or provide `source_workspace`; an explicit
scope must agree with the URL. Optional `instruction` (max 2 000 chars) is stored on the job and sidecar; U2Net does not consume it. Poll
`GET /api/v1/status/{job_id}` and cancel with `POST /api/v1/cancel/{job_id}`.

```bash
curl -X POST "$HOCUSPOCUS_URL/api/v1/tools/remove-background" \
-H "Content-Type: application/json" \
-d '{
"asset_id": "asset_image_123",
"workspace": "default",
"instruction": "preserve the hair edges",
"provenance": {"actor": "user"}
}'
```
Expand All @@ -728,22 +731,51 @@ transparent-PNG technical metadata.

## Tools upscale

`POST /api/v1/tools/upscale` is one shared post-processing action for either a
still image or a video. Send `{ "source": "image.png", "source_kind":
Studio and Wizard upscale now submit version 2 `tools.upscale` through
`POST /api/v1/generation/commands`; MCP exposes the same operation by name.
The shared contract requires an explicit method, source kind, output workspace,
and an asset ID or canonical local media URL as `input.params.source`.
It rejects host paths and bare filenames. See the
[shared command guide](../../docs/development/SHARED_NATIVE_COMMANDS.md) for
the envelope, receipts and replay, and
[Tools commands](../../docs/development/TOOLS_COMMANDS.md) for source identity.

The native legacy `POST /api/v1/tools/upscale` handles either a still image
or a video. Send `{ "source": "image.png", "source_kind":
"image", "asset_id": "...", "source_workspace": "...", "method":
"flashvsr2", "workspace": "default" }` for an image, or keep the legacy
`video_path` field with `source_kind: "video"` for a clip. Supported image
`video_path` field with `source_kind: "video"` for a clip. Built-in spatial
methods include `flashvsr2` (native default), `flashvsr3`, `flashvsr4`,
`flashvsr2pass2`, `flashvsr2pass4`, `lanczos1.5`, and `lanczos2`. Additional
native processors have media/platform constraints; discover the current schema
through `GET /api/v1/generation/commands` and processor availability in the
Tools panel. Conflicting `source` / `source_path` / `video_path` values return
`409`. Supported image
formats are `.bmp`, `.gif`, `.jpeg`, `.jpg`, `.png`, `.tif`, `.tiff`, and
`.webp`; supported video formats are `.avi`, `.m4v`, `.mkv`, `.mov`, `.mp4`,
`.mpeg`, `.mpg`, `.webm`, and `.wmv`. The source must be an exact asset, upload, or
file inside the selected workspace roots; path traversal and mismatched asset
IDs/kinds are rejected. Images use the existing spatial upsampler in still
mode and produce a new PNG beside the source. Videos retain the existing
mode and produce a new PNG in the destination workspace. Videos retain the existing
audio-preserving pipeline and produce a new video. Neither path overwrites its
source. Poll the returned job with `GET /api/v1/status/{job_id}` and cancel it
with `POST /api/v1/cancel/{job_id}`. Activity and the canonical asset manifest
retain the source lineage, method, workspace, provenance, and execution mode.

## Tools revoice

`POST /api/v1/tools/revoice` replaces voices on an existing **video** with
SeedVC. Body: `{ "video_path": "take.mp4", "voice_ref_paths": ["ref.wav"],
"mode": "single"|"two", "diffusion_steps": 25, "cfg_rate": 0.5,
"workspace": "default" }`. At least one and at most two reference paths are
required (audio or video). Supply two references for `mode: "two"`; with one,
the worker falls back to single-voice conversion. `mode` defaults to `single`
and any other string is coerced to `single`. The worker copies the source to a new `_revoiced`
file, then converts the copy; the original clip is never mutated. Failure
when the clip has no audio or SeedVC is unavailable. Same poll/cancel
endpoints as the other Tools jobs. See
[`docs/tools/HOWUSEIT.md`](../../docs/tools/HOWUSEIT.md).

## Gallery mix kinds

`GET /api/v1/outputs` accepts `result_kind=music_video|trailer|series_episode` (plus the existing `media_type`, `multiclip_only`, `favorites_only`, `search`, `workspace`, `limit`, `offset`). Classification lives in `services.output_result_kind` and applies only to **assembled** filenames (`multiclip`, `_mv.mp4`, `_movie.mp4`, `_rejoin_multiclip.mp4`, `_series_assembly`). Requesting `series_episode` also matches `chapter`. When `result_kind` is set, pagination is bypassed and every match is returned.
Expand Down Expand Up @@ -835,4 +867,4 @@ These routes always use the server active output folder. They do not accept `?wo
- `POST /api/v1/director/pipeline/{pid}/resume` and `POST /api/v1/director/pipeline/{pid}/continue` use the singular `pipeline` path.
- Batch prompt rewrite is UI-only: loop `POST /api/v1/llm/generate` (local LLM) then PUT the chosen prompts.

Operator notes: `docs/video-editor/HOWUSEIT.md`, `docs/workspaces/HOWUSEIT.md`, and `docs/character-kits/HOWUSEIT.md`.
Operator notes: `docs/tools/HOWUSEIT.md`, `docs/video-editor/HOWUSEIT.md`, `docs/workspaces/HOWUSEIT.md`, and `docs/character-kits/HOWUSEIT.md`.
2 changes: 2 additions & 0 deletions docs/HOWUSEIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Operator guides for HocusPocus subsystems. Prefer these over inventing a second workflow.

- **Studio Tools** (upscale, SeedVC revoice, rembg background removal — new file, never overwrite): [`docs/tools/HOWUSEIT.md`](tools/HOWUSEIT.md)
- **Video Editor** (import Lab clips, timeline, export) and **assembled mixes** (`result_kind` gallery tabs): [`docs/video-editor/HOWUSEIT.md`](video-editor/HOWUSEIT.md)
- **Workspaces tab** (Director generation threads — not the output-folder selector): [`docs/workspaces/HOWUSEIT.md`](workspaces/HOWUSEIT.md)
- **3D Video compositor** (Hunyuan meshes + plates + camera + rain/fog, mixed with MiniMax H3): [`docs/3d-video-compositor/HOWUSEIT.md`](3d-video-compositor/HOWUSEIT.md)
- **Character Kits / Face Rig** (2D cutout puppets, mouth overlays, dialogue cadence): [`docs/character-kits/HOWUSEIT.md`](character-kits/HOWUSEIT.md)
4 changes: 3 additions & 1 deletion docs/character-kits/HOWUSEIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ UI: **3D Video** sidebar (`SceneAnimatorPanel` → Character Kits). Code:
`app/_launch_runtime.py`.

Related: [3D Video compositor](../3d-video-compositor/HOWUSEIT.md),
[Character Creator orbit](../3d-video-compositor/HOWUSEIT.md#54-hunyuan3d-mesh).
[Character Creator orbit](../3d-video-compositor/HOWUSEIT.md#54-hunyuan3d-mesh),
[Studio Tools rembg](../tools/HOWUSEIT.md) (general image background removal;
Face Rig cleanup is a different endpoint).

---

Expand Down
218 changes: 218 additions & 0 deletions docs/tools/HOWUSEIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# HOWUSEIT — Studio Tools (upscale, revoice, remove background)

Operator guide for **post-processing existing media**. Tools do not invent a
new shot: they take one exact image or clip, write a **new** file, and leave
the source untouched.

UI: Studio sidebar → **Tools** (`generationMode: tools`,
`ui/src/components/Sidebar/ToolsPanel.tsx`). Upscale uses the version 2
`tools.upscale` operation at `POST /api/v1/generation/commands` from Studio,
Wizard and MCP. Revoice and remove-background still use
`POST /api/v1/tools/revoice` and `POST /api/v1/tools/remove-background`.
The native `/api/v1/tools/upscale` route remains for legacy clients. Workers:
`app/services/tools_upscale.py`, `app/shared/tools/`,
`app/_launch_runtime.py`. Poll and cancel with the shared job endpoints.

Related: [Video Editor](../video-editor/HOWUSEIT.md) (cut, do not regenerate),
[Character Kits Face Rig cleanup](../character-kits/HOWUSEIT.md)
(`POST /api/v1/character-kits/face-rig/cleanup` is a different rembg path),
[HTTP API](../../app/docs/API.md).

---

## 1. What this system is

| Tool | Accepts | Backend | Output |
|---|---|---|---|
| **Upscale / processors** | Image or video, depending on processor | FlashVSR, Lanczos and available native processors | New `_upscaled` PNG or video |
| **Revoice** | Video + 1–2 voice refs | SeedVC | New `_revoiced` clip (same container) |
| **Remove background** | Image | rembg U2Net | New `{stem}.no-background-{id}.png` |

Use Tools when the pixels (or voices) are already good and you need a
derivative. Use Studio generate when you need a new image/clip. Use Video
Editor when you only need trim, order, and export.

The three actions share the generation GPU slot, Activity footer, and
`GET /api/v1/status/{job_id}` / `POST /api/v1/cancel/{job_id}`. They are not
a second scheduler.

### Output folder versus Workspace collection

`workspace` on these routes is the **physical output folder**
(`default` or `[A-Za-z0-9][A-Za-z0-9_-]*`). It is not a logical Workspace
collection ID. Uploads use the virtual scope `__uploads__`. See
[domain model](../development/DOMAIN_MODEL_AND_ASSET_PROVENANCE.md).

---

## 2. Hard limits

1. **Never overwrite.** Every tool writes a new filename. A cancelled job
deletes its partial output when the worker can settle the cancel.
2. **Exact source.** For version 2 upscale, put an exact asset ID or canonical
local API media URL in `input.params.source`. Host paths, remote URLs and
bare filenames are rejected. Native Tools routes also accept confined
filenames/paths and a separate `asset_id`; do not mix those schemas.
3. **Source folder.** Keep the source folder distinct from the destination
`workspace`. A file URL carries it in `?workspace=`; an explicit
`source_workspace` must agree. Select a scope when an asset has multiple
locations. Upload URLs use `__uploads__`. Preserve the full canonical URL.
4. **Conflicting aliases.** The native upscale/revoice endpoints accept
`source`, `source_path`, and legacy `video_path`. Different values return
`409`. The version 2 upscale contract accepts only `params.source`.
5. **Kind gates.** Revoice is video-only. Remove-background is image-only
(`.png`, `.jpg`, `.jpeg`, `.webp`). Upscale images also allow
`.bmp`, `.gif`, `.tif`, `.tiff`. Videos:
`.avi`, `.m4v`, `.mkv`, `.mov`, `.mp4`, `.mpeg`, `.mpg`, `.webm`, `.wmv`.
6. **Instruction is metadata.** Remove-background accepts `instruction`
(max 2 000 chars) and stores it on the job/sidecar. U2Net does **not**
read it; the matte is the same with or without the note.

---

## 3. Operator workflow

1. Open Studio and choose **Tools** (Direct generation → Tools).
2. Pick **Upscale**, **Revoice**, or **Remove background**.
3. Set the source:
- gallery card → **Use selected gallery image/clip**
- resource selector: browse the library and confirm with **Choose**;
**Cancel** keeps the current selection
- upload through the same source field (image/video as allowed by the tool)
- from a selected video: **Send to Tools** / quick upscale in the info bar
4. Set tool-specific parameters. Run. Upscale presents the prepared request
before admission and returns a shared receipt. Watch the footer; the gallery
refreshes on `completed`. Failed/cancelled tiles remain available to inspect.

Wizard has adapters for upscale and remove-background. It does not currently
expose a dedicated Revoice action; use the Tools panel for Revoice. MCP has the
shared `tools.upscale` operation; that entry does not imply shared operations
for all three tools. See [shared commands](../development/SHARED_NATIVE_COMMANDS.md).

---

## 4. Upscale

Built-in spatial choices include:

```
flashvsr2, flashvsr3, flashvsr4, flashvsr2pass2, flashvsr2pass4,
lanczos1.5, lanczos2
```

Additional processor choices are discovered from the server and filtered by
source kind and availability. Consult `GET /api/v1/generation/commands` for
the `tools.upscale` schema and the panel's processor options; the list above
is not the whole catalog. The shared command requires an explicit `method`.
Only the native legacy route defaults an omitted method to `flashvsr2`.

FlashVSR is model-based super-resolution (weights download on first use).
Lanczos is a fast classic resize. If Settings → Services has FlashVSR mode
`0`, the panel warns but still lets you pick a FlashVSR method.

- Images go through the spatial upsampler in still mode and always become a
new PNG (`_upscaled`). Optional `seed` is an integer (`-1` default).
- Videos keep the existing audio-preserving pipeline and write a new clip
(`_upscaled` + configured container).

```bash
curl -X POST "$HOCUSPOCUS_URL/api/v1/generation/commands" \
-H "Content-Type: application/json" \
-d '{
"version": 2,
"operation": "tools.upscale",
"intent_id": "upscale-still-001",
"input": {
"workspace": "default",
"params": {
"source": "/api/v1/file/still.png?workspace=default",
"source_kind": "image",
"method": "lanczos2"
}
}
}'
```

Replace the example source with an existing resource. Keep the same
`intent_id` when retrying an uncertain response to this request; choose a new
one for another intentional operation. Admission returns `receipt.result.job_id`
and `receipt.result.task_id`; it does not mean the file is complete. Follow the
job status or recover through `generation.receipt` as described in
[shared commands](../development/SHARED_NATIVE_COMMANDS.md).

Native legacy clients may keep using `/api/v1/tools/upscale` with the flat
body and `video_path` alias. That endpoint does not provide the shared
command receipt/replay contract; see [Tools command contract](../development/TOOLS_COMMANDS.md).

---

## 5. Revoice (SeedVC)

Body: `{ video_path, voice_ref_paths: [1–2 paths], mode?: "single"|"two",
diffusion_steps?: 25, cfg_rate?: 0.5, workspace? }`.

| Mode | Effect |
|---|---|
| `single` (default) | Replace every voice with the first reference |
| `two` | Detect two speakers; first → Voice A, second → Voice B; keep music and silence |

Supply two references for `two`; with only one, the worker falls back to
single-voice conversion. Any other `mode` string is coerced to `single`.
Voice refs may be audio or
video files resolved inside the destination folder or uploads. The worker
copies the source first, then converts the copy.

Failure cases you will actually see: clip has no audio, SeedVC is
unavailable, or no reference file could be resolved.

```bash
curl -X POST "$HOCUSPOCUS_URL/api/v1/tools/revoice" \
-H "Content-Type: application/json" \
-d '{
"video_path": "take.mp4",
"voice_ref_paths": ["ref-a.wav"],
"mode": "single",
"workspace": "default"
}'
```

---

## 6. Remove background

Prefer `asset_id`. `source` alone is accepted. Destination `workspace`
defaults to the server active output folder.

```bash
curl -X POST "$HOCUSPOCUS_URL/api/v1/tools/remove-background" \
-H "Content-Type: application/json" \
-d '{
"asset_id": "asset_image_123",
"workspace": "default",
"provenance": {"actor": "user"}
}'
```

Accepted immediately with `job_id`, `task_id`, `root_task_id`, and frozen
`generation_details.model_type: rembg-u2net`. The sidecar records source
lineage, timings, and transparent-PNG metrics (`width`, `height`, `alpha`).

Face Rig overlay cleanup is a **different** endpoint
(`POST /api/v1/character-kits/face-rig/cleanup`) that also uses rembg U2Net
plus crop-to-alpha. Do not substitute one for the other.

---

## 7. Pitfalls

- Sending a `.bmp` / `.tif` to remove-background fails; those formats are
upscale-only.
- Revoice on an image is rejected by both the panel and the HTTP
`expected_kinds=("video",)` gate.
- `instruction` will not “preserve hair.” It is stored, not consumed.
- Two different `source` / `video_path` values fail with `409`, not a silent
pick-one.
- Gallery URLs look like `/api/v1/file/clip.mp4?workspace=default`. Keep the
query: it identifies the source folder. It is not part of the disk filename.
- Tools share the GPU lock with Studio generate. A long FlashVSR job blocks
the next generation until it finishes or is cancelled.
2 changes: 1 addition & 1 deletion docs/video-editor/HOWUSEIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This is not MiniMax H3 and not the 3D compositor. The editor **never regenerates

UI tab: **Video Editor** (`mediaFilter: videoeditor`). Code: `ui/src/features/video-editor/`. Render: `app/services/video_editor.py`. HTTP: `app/_launch_runtime.py`. Mix kinds: `app/services/output_result_kind.py`.

Related: [3D Video compositor](../3d-video-compositor/HOWUSEIT.md) §5.8, [Workspaces / Director threads](../workspaces/HOWUSEIT.md).
Related: [Studio Tools](../tools/HOWUSEIT.md) (upscale / revoice / rembg), [3D Video compositor](../3d-video-compositor/HOWUSEIT.md) §5.8, [Workspaces / Director threads](../workspaces/HOWUSEIT.md).

---

Expand Down