add models

.claude/skills/mind-benchmark/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: mind-benchmark
+description: "Benchmark an interactive world model (Waypoint, ABot, matrix, dreamx, sana, ...) on the MIND benchmark at C:\\workspace\\world\\MIND. Use when driving a model over MIND, scoring memory-consistency/action-control metrics, adding a new model driver, running the mirror test, or comparing models. Covers the drive->score->compare flow and its Windows/PowerShell gotchas."
+---
+
+# MIND world-model benchmarking
+
+MIND (`C:\workspace\world\MIND`) scores how well an interactive world model keeps **memory consistency** and **action control**. GT is at `C:\workspace\world\MIND-Data` (`{1st_data,3rd_data}/test/{action_space_test,mem_test,mirror_test}`). Driven videos go to `C:\workspace\world\MIND-tests\<model>\...`. Scores are `result_<model>_<timestamp>.json` in the MIND root.
+
+## The pipeline (3 stages)
+1. **Drive** a model on MIND actions -> videos in `MIND-tests\<model>`.
+2. **Score** vs GT -> `result_<model>_*.json`.
+3. **Compare** via the scores table.
+
+```
+.\drive_<model>.bat --limit 5        :: smoke first
+.\drive_<model>.bat                  :: full (both perspectives)
+.\drive_<model>.bat --mirror-test    :: mirror clips (separately)
+.\run_mind.bat --% <model> "" 1 both :: score all 5 metrics ("" = default)
+.\scores.bat <model>                 :: view newest result table
+```
+
+## Metrics (arrows = better direction)
+`lcm`↑ (memory consistency, PSNR-like), `visual`↑ (quality), `dino`↓ (semantic drift), `avg_mse`↓ (pixel err), `gsc`↓ (mirror consistency — splits clip, time-flips return vs outbound; lower=you returned to the same view), `action` (control). Default = all 5. `gsc` needs mirror clips; `action` needs ViPE (JIT-compiles a CUDA ext first use; `run_mind.bat` sets up vcvars + CUDA_HOME) and is SLOW (~300 s/video vs ~15 s for the other 4).
+- **`action` often writes nothing (n/a) even when requested** — a known scoring bug; the other 5 metrics still give a clear verdict. Don't block a comparison on it.
+- Result JSON = `{data:[{path, perspective, test_type, lcm{}, visual_quality{}, dino{}, ...}], ...}`; metrics are nested dicts, so reduce via `_scores_table.py`, not by hand.
+
+## Reading / splitting results
+`scores.bat <model>` shows the newest result (aggregate over both perspectives). To split by perspective, filter `data` to `perspective=="1st_data"`/`"3rd_data"` into a temp JSON and run `_scores_table.py` on each. The `150/100` sample split in a full run = 150 action_space+mem (lcm/visual/dino/mse) + 100 mirror (gsc).
+
+## Gotchas (these bite every time)
+- **PowerShell comma trap:** `run_mind.bat waypoint lcm,visual,dino ...` fails ("4th arg ... got: dino") because cmd splits commas. Fix: `run_mind.bat --% <model> "lcm,visual,dino" 1 both` (`--%` AND quotes), or just pass `""` to get the all-5 default.
+- **Resume adds samples, NOT metrics:** if a model was scored with fewer metrics, `run_mind` **skips** those samples forever. To add `action`/`gsc`, `del result_<model>_*.json` then re-score fresh.
+- **`.bat` in PowerShell needs `.\`**; `.bat` files must be CRLF.
+- **CPU torch:** the MIND venv can silently get `torch+cpu` -> `process.py` sees 0 GPUs -> `mp.Pool(0)` crash. Reinstall the **ViPE-compatible** CUDA build (see below), NOT an arbitrary one.
+- **Free the GPU** before scoring/driving (shared 32 GB); check `nvidia-smi` first.
+
+## action / ViPE (the fragile metric)
+`action` shells out to a **`vipe` CLI** (built into the MIND venv via `build_vipe.bat`). It is PINNED to a torch/ABI:
+- MIND venv must be **torch 2.10.0 + cu128** (has CUDA + sm_120 for the 5090) + **flash_attn 2.8.3+cu128torch2.10-cp310** (mjun0812 prebuilt wheel, `--no-build-isolation`). Then `build_vipe.bat` (needs MSVC/vcvars + cl.exe) produces `.venv\Scripts\vipe.exe`.
+- **`build_vipe.bat`'s `CUDA_HOME` must match torch's CUDA.** For torch cu128 set `CUDA_HOME=...\CUDA\v12.8` (NOT v13.0). Wrong version -> `RuntimeError: detected CUDA version mismatches` during the native build. The bat had v13.0 hardcoded — check line ~33 matches your torch.
+- **Reinstalling a different torch (e.g. cu130 / 2.13) WIPES the ViPE build** -> `action` crashes per-sample with `ViPE extract_traj: FileNotFoundError [WinError 2]` (the `vipe` exe is gone) and silently writes n/a. This exact mistake cost hours. If you must fix CPU-torch, reinstall **torch==2.10.0 --index-url .../cu128**, re-add the flash_attn wheel, then re-run `build_vipe.bat`.
+- The editable build does NOT pull ViPE's runtime deps -> `vipe.exe` crashes with `ModuleNotFoundError: No module named 'hydra'`. Fix: `uv pip install --python .venv\Scripts\python.exe hydra-core` (install any other missing runtime deps the CLI reports too).
+- Verify before scoring with action: `.venv\Scripts\vipe.exe --help` should list `infer`/`visualize` (no traceback).
+- Full fix chain: torch 2.10/cu128 -> flash_attn wheel (`--no-build-isolation`) -> CUDA_HOME=v12.8 -> `build_vipe.bat` -> `hydra-core` -> `vipe.exe --help` works.
+- `action` is ALSO slow (~300 s/video vs ~15 s for the other 4). For a fast comparison, exclude it: `"lcm,visual,dino,gsc"`.
+
+## Adding a new model driver
+Model runs in ITS OWN venv/project. Write `src\drive_<model>.py` + `drive_<model>.bat` (runs via that model's python). Pattern:
+- `gather()` walks `{persp}/test/{action_space_test,mem_test}/<name>/{video.mp4,action.json}`; output `MIND-tests\<model>\{persp}\{test_type}\<name>\video.mp4`; skip-if-exists.
+- **Convert MIND actions** (`ws/ad/ud/lr` tri-state: 0=neutral,1/2=dirs) to the model's control format. Examples: Waypoint (world_engine) -> Windows VK codes (W=0x57,S=0x53,A=0x41,D=0x44) + mouse for lr/ud. ABot -> `{keys:{W,A,S,D,I,J,K,L}}` (WASD move, IJKL look; lr->J/L, ud->I/K).
+- **Mirror test:** `from utils.mirror_test_utils import gather_mirror_samples, MIRROR_DEFAULT_ACTION`; seed each `data-NN.png` (a PNG, not a video) + replay the `-w.json` go-then-return; output under `mirror_test`.
+- **LOAD-ONCE for speed:** never subprocess-per-sample (reloads the model each time). Either loop in-process (Waypoint keeps the engine loaded), or build a manifest + one call to a batch mode (ABot's `inference.py --mind-batch`, which loops the lru_cached pipeline). Quantize for more speed (ABot `--quant fp8-per-tensor` on Blackwell).
+
+## Existing drivers
+`drive_waypoint.bat` (world_engine, scope-overworld venv), `drive_abot.bat` (ABot venv, `--mind-batch` load-once + FP8), plus `drive_matrix3/dreamx/sana/lingbot/deepverse.bat` (each needs its backend installed). `drive_all_with_mirror.bat` runs the default set. `scores.bat` shows the newest result; `scores.bat` (no arg) = all models.

.gitignore

@@ -0,0 +1,64 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.egg
+*.egg-info/
+.eggs/
+build/
+dist/
+wheels/
+*.whl
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# IDE / editor
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+Thumbs.db
+
+# Logs and caches
+*.log
+.cache/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# Build artifacts
+build_vipe.log
+
+# Model weights / downloads
+dinov3_vitb16/
+*.safetensors
+*.bin
+*.pt
+*.pth
+*.ckpt
+*.onnx
+
+# Data / outputs
+outputs/
+output/
+results/
+results_bak/
+result_*.json
+runs/
+checkpoints/
+*.mp4
+*.avi
+*.mov
+*.mkv
+
+# Local / scratch
+scratch/
+tmp/
+temp/

_scores_table.py

@@ -0,0 +1,125 @@
+"""Extract MIND result scores in the lcm/gsc/visual/dino/action/avg_mse layout.
+
+Schema (discovered by probing existing result JSONs):
+  data[i].lcm            -> {avg_mse, avg_psnr, avg_ssim, avg_lpips, length, ...}
+  data[i].gsc            -> {avg_gsc?, ...}                # path depends on metric version
+  data[i].visual_quality -> {avg_imaging, avg_aesthetic, ...}
+  data[i].dino           -> {avg_dino_mse, ...}
+  data[i].action         -> {__overall__: {rpe_trans_mean, rpe_rot_mean_deg}, ...}
+
+Each entry in `data` is one staged sample. We mean across samples.
+"""
+import io
+import json
+import statistics
+import sys
+from pathlib import Path
+
+sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", line_buffering=True)
+
+
+def vals(data, metric, key):
+    out = []
+    for entry in data:
+        m = entry.get(metric)
+        if isinstance(m, dict):
+            v = m.get(key)
+            if isinstance(v, (int, float)):
+                out.append(v)
+    return out
+
+
+def gsc_vals(data):
+    """gsc lives nested at data[i].video_results[0].gsc.mse (a per-frame mse list).
+    Per-sample summary = mean of that list. Returns the list of per-sample means."""
+    import statistics as _st
+    out = []
+    for entry in data:
+        for vr in entry.get("video_results", []) or []:
+            g = vr.get("gsc")
+            if isinstance(g, dict):
+                mse = g.get("mse")
+                if isinstance(mse, list) and mse:
+                    out.append(_st.mean(mse))
+                break  # first video_results entry only
+    return out
+
+
+def overall(data, metric, key):
+    out = []
+    for entry in data:
+        m = entry.get(metric)
+        if isinstance(m, dict) and isinstance(m.get("__overall__"), dict):
+            v = m["__overall__"].get(key)
+            if isinstance(v, (int, float)):
+                out.append(v)
+    return out
+
+
+def first_dict(data, metric):
+    """Find any non-empty `metric` dict to probe its keys."""
+    for e in data:
+        m = e.get(metric)
+        if isinstance(m, dict) and m:
+            return m
+    return None
+
+
+def find_avg_key(metric_obj, candidates):
+    """Pick the first matching avg_* key from a list of candidates."""
+    if not metric_obj:
+        return None
+    for c in candidates:
+        if c in metric_obj:
+            return c
+    return None
+
+
+def mean(xs):
+    return statistics.mean(xs) if xs else None
+
+
+def fmt(v):
+    return "    n/a" if v is None else f"{v:>9.4f}"
+
+
+def main():
+    path = sys.argv[1] if len(sys.argv) > 1 else None
+    if not path:
+        print("usage: _scores_table.py <result_*.json>", file=sys.stderr)
+        sys.exit(2)
+    d = json.load(open(path, encoding="utf-8"))
+    data = d.get("data", [])
+    print(f"file:    {Path(path).name}")
+    print(f"samples: {len(data)}")
+    print()
+
+    # Per-metric: try a list of candidate key names so this works across MIND
+    # versions that have evolved key naming.
+    lcm_obj = first_dict(data, "lcm")
+    gsc_obj = first_dict(data, "gsc")
+    vis_obj = first_dict(data, "visual_quality")
+    dno_obj = first_dict(data, "dino")
+
+    lcm_psnr_key = find_avg_key(lcm_obj, ["avg_psnr", "psnr_mean", "psnr"])
+    lcm_mse_key  = find_avg_key(lcm_obj, ["avg_mse",  "mse_mean",  "mse"])
+    gsc_key      = find_avg_key(gsc_obj, ["avg_gsc", "gsc_mean", "gsc"])
+    vis_key      = find_avg_key(vis_obj, ["avg_imaging", "avg_musiq", "musiq_mean"])
+    dno_key      = find_avg_key(dno_obj, ["avg_dino_mse", "dino_mse_mean", "avg_dino"])
+
+    rows = [
+        ("lcm ↑",     vals(data, "lcm",            lcm_psnr_key) if lcm_psnr_key else []),
+        ("gsc ↓",     gsc_vals(data)),  # gsc is MSE-based → lower is better
+        ("visual ↑",  vals(data, "visual_quality", vis_key)      if vis_key      else []),
+        ("dino ↓",    vals(data, "dino",           dno_key)      if dno_key      else []),
+        ("action",    overall(data, "action", "rpe_trans_mean")),
+        ("avg_mse ↓", vals(data, "lcm",            lcm_mse_key)  if lcm_mse_key  else []),
+    ]
+    print(f"  {'metric':<12}  {'mean':>9}   n")
+    print("  " + "-" * 30)
+    for label, xs in rows:
+        print(f"  {label:<12}  {fmt(mean(xs))}   {len(xs):>3}")
+
+
+if __name__ == "__main__":
+    main()

_smoke.py

@@ -0,0 +1,15 @@
+import torch
+import lpips
+import torchmetrics
+import pyiqa
+import clip
+import transformers
+import av
+import huggingface_hub
+
+print(f"torch={torch.__version__}  cuda_available={torch.cuda.is_available()}")
+print(f"transformers={transformers.__version__}")
+print(f"huggingface_hub={huggingface_hub.__version__}")
+print(f"pyiqa={pyiqa.__version__}")
+print(f"lpips={lpips.__version__}")
+print(f"clip OK; torchmetrics OK; av OK")

build_vipe.bat

@@ -0,0 +1,125 @@
+@echo off
+REM build_vipe.bat - build ViPE in-place inside MIND's existing venv.
+REM
+REM What this does:
+REM   1) Loads MSVC build env via vcvars64.bat (so cl.exe is on PATH).
+REM   2) Resolves MIND's .venv python (torch 2.10.0+cu128, sm_120 in arch_list).
+REM   3) Ensures matching flash-attn prebuilt wheel is installed (skipped if present).
+REM   4) Runs `pip install --no-build-isolation -e .` from MIND\vipe to compile
+REM      and link ViPE's native CUDA extensions in-place with sm_120 support.
+REM
+REM Key compile-flag detail: ViPE's specs.py passes -DUSE_CUDA so torch >= 2.10's
+REM compiled_autograd.h Windows guard fires and skips an if-constexpr block that
+REM otherwise triggers MSVC C2872 'std: ambiguous symbol' when Eigen is also in
+REM the include chain.
+REM
+REM ViPE's setup.py downloads Eigen 3.4 automatically (USE_SYSTEM_EIGEN=0 by
+REM default), so we don't need a separate Eigen install.
+REM
+REM Prerequisites:
+REM   - Visual Studio 2022 Community/Pro with "Desktop development with C++".
+REM   - CUDA Toolkit 12.x on PATH (nvcc.exe). torch 2.10.0+cu128 needs CUDA 12.x.
+REM
+REM Usage:
+REM   .\build_vipe.bat              (build)
+REM   .\build_vipe.bat /clean       (clear ViPE build cache then build)
+
+setlocal enableextensions enabledelayedexpansion
+
+set "PY=%~dp0.venv\Scripts\python.exe"
+set "VIPE_DIR=%~dp0vipe"
+set "VS_VCVARS=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
+REM Pin CUDA Toolkit to 13.0 (matches torch 2.12+cu130 now in the MIND venv).
+set "CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8"
+set "CUDA_PATH=%CUDA_HOME%"
+set "PATH=%CUDA_HOME%\bin;%CUDA_HOME%\libnvvp;%PATH%"
+REM TORCH_CUDA_ARCH_LIST: include sm_120 so RTX 5090 SASS is baked into vipe_ext.pyd.
+set "TORCH_CUDA_ARCH_LIST=8.0;8.6;9.0;10.0;12.0"
+
+if not exist "%PY%" (
+    echo ERROR: MIND venv python not found: %PY%
+    echo Create the venv with `uv venv` from this directory first.
+    exit /b 1
+)
+if not exist "%VIPE_DIR%\setup.py" (
+    echo ERROR: ViPE source not found at %VIPE_DIR%
+    exit /b 1
+)
+if not exist "%VS_VCVARS%" (
+    echo ERROR: vcvars64.bat not found at %VS_VCVARS%
+    echo Adjust VS_VCVARS at the top of this script to your VS 2022 install path.
+    exit /b 1
+)
+if not exist "%CUDA_HOME%\bin\nvcc.exe" (
+    echo ERROR: CUDA 12.8 nvcc not found at %CUDA_HOME%\bin\nvcc.exe
+    echo Adjust CUDA_HOME at the top of this script if your toolkit is elsewhere.
+    exit /b 1
+)
+
+if /i "%~1"=="/clean" (
+    echo Clearing ViPE build cache...
+    if exist "%VIPE_DIR%\build" rmdir /s /q "%VIPE_DIR%\build"
+    if exist "%VIPE_DIR%\vipe.egg-info" rmdir /s /q "%VIPE_DIR%\vipe.egg-info"
+    for /f "delims=" %%D in ('dir /s /b /ad "%VIPE_DIR%\__pycache__" 2^>nul') do rmdir /s /q "%%D"
+    shift
+)
+
+echo.
+echo === Loading MSVC build env (vcvars64.bat) ===
+call "%VS_VCVARS%" >nul
+if errorlevel 1 (
+    echo [build_vipe] vcvars64.bat failed.
+    exit /b %ERRORLEVEL%
+)
+where cl >nul 2>&1 || (
+    echo [build_vipe] cl.exe still not on PATH after vcvars; MSVC env load failed.
+    exit /b 1
+)
+
+echo.
+echo === Tool sanity ===
+where nvcc 2>nul && nvcc --version | findstr /R "release"
+"%PY%" --version
+"%PY%" -c "import torch; print('torch', torch.__version__, 'cuda', torch.version.cuda); print('arch_list:', torch.cuda.get_arch_list())"
+"%PY%" -c "import flash_attn; print('flash_attn', flash_attn.__version__)" || (
+    echo [build_vipe] flash_attn import failed.
+    echo Install a prebuilt wheel matching torch + cu128 + cp310 + win, e.g.:
+    echo   uv pip install --python "%PY%" --no-build-isolation https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.13/flash_attn-2.8.3+cu128torch2.10-cp310-cp310-win_amd64.whl
+    exit /b 1
+)
+
+echo.
+echo === Ensuring in-venv build deps (wheel/setuptools/ninja) for --no-build-isolation ===
+uv pip install --python "%PY%" wheel setuptools ninja
+
+echo.
+echo === Building ViPE editable into MIND venv ===
+echo (Compiles native CUDA extensions with sm_120 + USE_CUDA; first build ~5-10 min.)
+pushd "%VIPE_DIR%"
+uv pip install --python "%PY%" --no-build-isolation -e .
+set "RC=%ERRORLEVEL%"
+popd
+if not "%RC%"=="0" (
+    echo [build_vipe] ViPE native build failed with exit code %RC%.
+    echo Common causes:
+    echo   - MSVC C2872 'std: ambiguous' = USE_CUDA define missing ^(see specs.py^).
+    echo   - flash_attn wheel ABI mismatch with installed torch.
+    echo   - Out of disk space in TEMP during Eigen download.
+    exit /b %RC%
+)
+
+echo.
+echo === Verifying ViPE import + CLI ===
+"%PY%" -c "import vipe; print('vipe package:', vipe.__file__)" || (
+    echo [build_vipe] post-build import of vipe failed.
+    exit /b 1
+)
+
+echo.
+echo ============================================================
+echo build_vipe: done. ViPE installed editable into MIND venv:
+echo   %PY%
+echo You can now run scoring with action enabled:
+echo   .\run_mind.bat ^<test_subdir^> lcm,visual,dino,action,gsc
+echo ============================================================
+exit /b 0