How the pieces fit, file by file, and what each stage reads and writes. The skills in skills/ say how to run it;
this says what's inside.
Every stage agrees on one 1280 x 720 layout (mocap/layout.py, mirrored in blender/common.py):
- Front view, left,
W_FRONTpx wide: an orthographic camera at -Y looking along +Y at the creature's face. Image u grows with world X. - Side view, right,
W_SIDEpx wide: an orthographic camera at -X looking along +X at the creature's right flank, head to the right. Image u grows with world -Y. - Both share the vertical axis: v grows downward with -Z, the ground (z = 0) is row
GROUND_V, and one pixel isM_PER_PXmetres in both views. A joint at height z sits on the same row in both halves. That is what makes two views enough for 3D (x from the front, y from the side, z from both) and what the sync check measures. - The studio: grey wall (128, 130, 134), a lighter floor band below the ground row (152, 154, 158), a dark divider.
MOCAP_CREATURE=assets/<c> selects a creature; its creature.json sets the layout, the paint and the feet. Without
it, every module falls back to the wolf's layout (420 + 860 px, 1.9 m of height in frame).
| file | written by | holds |
|---|---|---|
assets/<c>/model.glb |
Tripo or any generator | the textured mesh |
assets/<c>/creature.json |
you | prep (length_m, rotz_deg, axis), layout, paint (limb groups: colour + bones), feet, prompt words (name, noun, describe, colour_code, count, flank, limbs), the 3/4 showcase camera; optional: contacts (ground contact joints), testpose, start_pose (film A-pose bipeds with arms down), rigid (hard single-bone skinning), fix_front_swaps |
assets/<c>/rig.json |
you | joints (metres, -Y forward, +X left), bones [name, head, tail, parent], limits_deg per bone or stem |
assets/<c>/prep/static.blend |
blender/prep_mesh.py |
the mesh at real size, facing -Y, on the ground |
assets/<c>/rigged/rigged.blend |
blender/build_rig.py |
armature, skinned mesh, limb paint as a colour attribute (legpaint) |
assets/<c>/rigged/rig_info.json |
blender/build_rig.py |
rest matrices, parents, lengths, limits, and the keypoints (which bone end each joint is) for the solver |
| stage | code | reads | writes |
|---|---|---|---|
| reference frame | blender/render_ref.py, tools/compose_ref.py |
rigged.blend | runs/<c>/ref/ref.png |
| prompt | tools/make_prompt.py |
creature.json + one action sentence | prompts/<c>/<move>.txt |
| video | mocap/generate.py (fal queue client in mocap/fal.py) |
ref.png + prompt | runs/<c>/videos/<move>/s<seed>/<model>.mp4 |
| dataset | blender/render_dataset.py |
rigged.blend + rig_info.json | runs/<c>/dataset/<i>_{front,side}.png, <i>_meta.json (true joints in metres and layout px) |
| reader | mocap/model.py, mocap/train.py |
dataset | runs/<c>/reader.pt (state dict, joint names, held-out error) |
| read | mocap/detect.py |
clip + reader + ref.png | det.npz: front_uv, side_uv (T x J x 2, layout px), front_conf, side_conf, fps, names, calib |
| fit | mocap/solve.py |
det.npz + rig_info.json | motion.json: bones, trans, root_quat, euler (T x B x 3), joints (T x J x 3 m), reproj_px, sync (incl. paw_agreement) |
| apply | blender/apply_motion.py |
rigged.blend + motion.json | keyframed rig, joints.json (true joint positions from Blender), renders, animation.glb |
| compare | mocap/showcase.py |
clip + det.npz + renders | showcase.mp4 |
| score | tools/score_takes.py |
every take's motion.json + det.npz | runs/<c>/takes.csv |
| shape | tools/shape_pack.py, tools/shape_clip.py |
picks/<c>.json + motion.json |
runs/<c>/pack/clips/<Name>.json (trimmed, looped, root motion), shape_report.json |
| pack | blender/pack_glb.py |
rigged.blend + shaped clips | runs/<c>/pack/<c>.glb, the limb paint taken out |
| feet | tools/locomotion.py |
root-motion loops | runs/<c>/pack/locomotion.json |
| exam | mocap/exam.py |
rig_info.json | a random motion, its two-camera video, and a per-joint error score in cm |
| LoRA | tools/build_lora_set.py, mocap/mirror.py, mocap/lora.py |
captured motions + prompts | training clips, the fal run, lora.json |
DINOv2 ViT-B/14 with registers (facebook/dinov2-with-registers-base, Apache-2.0), fine-tuned end to end (backbone at
2e-5, head at 1e-3, AdamW, cosine schedule with warm-up). Each view is padded to a square with the backdrop grey and
resized to 448 x 448 (32 x 32 patches); the patch tokens go through a small conv head that upsamples to one 128 x 128
heatmap per joint, trained against Gaussian targets (sigma 2). Peaks are refined to sub-pixel from their neighbours.
Training images are laid on random backdrops and degraded like video; 4% of samples are held out.
mocap/solve.py redoes Blender's forward kinematics in torch (pose = parent pose x rest offset x Euler XYZ), with the
root free and every other bone's Euler angles squashed into its limits by a sigmoid. Before fitting it scores the
clip (paw agreement, height gap and best lag between views) and time-warps the front view onto the side view when
that cuts the height gap by 15% or more. Joints are triangulated (x from the front, y from the side, height weighted
0.25 front / 1 side by confidence), the root rotation comes from a weighted Kabsch fit of the points riding on the
root bone, and an analytic IK pose aimed at the triangulated joints is the starting point. The loss is a robust
(Cauchy) reprojection error in both views, its scale annealed from 40 px to 6 px over the first 60% of 1,500 Adam
steps, plus joint acceleration, angular velocity, a small pull to rest, and a ground term. A first fit finds where each
contact joint is planted and the ground's own velocity (on an on-the-spot clip, planted feet ride backwards together);
a second holds each plant still relative to the ground, and writes contacts into motion.json.
- Nothing pins planted feet after the fit, so they slide 1 to 2 cm a step. A short IK pass would remove it.
- Mirrored
_L/_Rbones share one local Z axis, so a shared sideways limit is wrong for one side (anymove-rig). - The wing skeleton doesn't follow how a video folds and sweeps a feathered wing (the eagle: 15-30 px off).
mocap/exam.py's per-group scores use the wolf's joint names; the mean, median and p95 work for any creature.tools/build_lora_set.pymirrors motions assuming a symmetric rig.