MoCap Studio

Local-first markerless motion capture for turning ordinary videos into inspectable 3D motion data, kinematics, exports, and benchmark reports.

MoCap Studio targets the hard middle ground between a research prototype and a reviewable software system: offline video processing, dual-camera reconstruction, local visualization, export bundles, and reproducible detector comparisons.

The real-time VS5 multi-camera path is restored for local experimentation: single-camera capture, camera-server mode, and master-coordinator stereo mode are available through the desktop GUI launcher. The strongest review/validation surface is still local/offline Python processing plus the local FastAPI job API, because those paths have the most repeatable benchmark coverage.

Architecture Philosophy: Python-First Profiling

This project uses a profiling-first architecture. Python remains the primary research and execution engine for video loading, pose inference, timestamp synchronization, stereo triangulation, stabilization, kinematics, and exporting. C++ acceleration is strictly optional and only introduced for mathematically proven bottlenecks identified via the internal profiler. This maximizes development speed, readability, and compatibility with the wider machine learning ecosystem without prematurely sacrificing performance.

Profiling-First Optimization: Streaming & Performance Ceiling

Following this philosophy, we conducted an optimization pass targeting processing bottlenecks. Our internal PipelineProfiler proved that upfront video frame loading (read_all_frames) caused a massive 12+ second I/O latency block and forced memory consumption to scale linearly with video length.

To solve this, we implemented a memory-efficient Monotonic Streaming Generator (stream_sync_pairs). This generator consumes frames directly off the disk based on TARGET_FPS requested timestamps (using a 30ms alignment tolerance), reducing the I/O load per pair to ~5.5 ms and breaking the memory scaling bottleneck.

Strict Metric Delineation: To prevent false performance claims or confusion, the pipeline permanently isolates and reports three distinct metrics:

1. Source Video FPS: Driven exactly by cv2.CAP_PROP_FPS.
2. Target Analysis FPS: Driven by the requested TARGET_FPS configuration.
3. Effective Pipeline FPS: Exclusively reported as processed frame pairs / wall-clock second.

Performance Ceiling (May 2026 Audit): A 50-frame dual-camera benchmark was executed under four configurations. The best measured CPU result on this hardware/configuration was recorded using the streaming generator combined with ThreadPoolExecutor parallel inference. Parallel inference uses separate PoseLandmarker instances for each camera stream and avoids shared mutable model state. Output equivalence is validated against sequential inference.

• Mode A (Sequential + Render + Buffered): 2.43 FPS
• Mode C (Sequential + No-Render + Streaming): 9.73 FPS
• Mode D (Parallel + No-Render + Streaming): 12.40 FPS

Note: We do not claim 15 FPS scalability; the benchmarked parallel streaming architecture provides a reliable ~12 FPS ceiling while solving the memory/IO constraints.

What Makes It Interesting

Capability	What exists now
Markerless pose pipeline	MediaPipe pose baseline with optional face/hand dense extraction
Dual-camera processing	front/right upload API, frame pairing, calibration-aware triangulation
Production-ish local API	chunked uploads, upload validation, bounded worker queue, export downloads
Stable exports	annotated video, metrics CSV, raw 3D nodes CSV, kinematics CSV, summary JSON
Benchmarks	MediaPipe baseline, optional MMPose/RTMPose aliases, dual-camera report writer
Real-time VS5 mode	live GUI launcher, camera server, master coordinator, timestamp sync, 1-Euro filtering, and live bone-stability diagnostics
Local visualization	raw node replay and local viewer workflows
Repo hygiene	CI, tests, formatted docs, archived historical milestones

Visual Preview

Architecture overview:

Pipeline overview:

Sample result plot:

System comparison:

Small curated outputs live under docs/results. Generated videos and full benchmark reports stay out of Git.

Before And After

Before	After
Raw video frames	Annotated motion video
Detector landmarks only	Kinematics CSV and raw 3D node CSV
One-off scripts	Local API with job states and export endpoints
Silent generated files	Structured results.json, summary.csv, and REPORT.md
Unclear experiments	Comparison sandbox with skipped/error states recorded
Huge pipeline function	Split API, job runner, video, model, triangulation, and export services

Validation Snapshot

For a detailed summary of our most recent validation pipelines (Markerless Calibration and EasyMocap Kinematics Benchmarking), please see:

• Validation and Benchmarking Summary (Deep dive)
• Validation Results Short Summary (Executive summary)

Check	Current status
Focused pytest suite	21 tests passing locally
Ruff lint	passing on backend/comparison/test targets
Black format check	passing on backend/comparison/test targets
Mypy	passing for comparison/backends
Real trial video discovery	found two local MP4 trial clips
Offline validation video 1	processed 348 frames; quality gates failed
Offline validation video 2	processed 281 frames; balanced and strict gates passed
Dual-camera MediaPipe benchmark	processed 281 paired frames; triangulation success 22.78%
May 16 dual-camera diagnosis	timestamp-synced MediaPipe lite/full/heavy sweep completed
Manual offset sweep	MediaPipe full, 150 paired frames, best tested offset +1000 ms
Optional RTMPose smoke	separate MMPose venv; t/s/m aliases, 100 paired frames each

The real trial snapshot is intentionally honest: offline_validation_benchmark_20260603.json. It reports detector coverage, stability, runtime, quality gates, and triangulation availability. It does not claim MPJPE or PA-MPJPE because no external ground truth is committed.

Benchmark Results

Offline MediaPipe Validation

Clip	Frames	Detected coverage	Usable coverage	FPS	Reliability	Gate
2026-05-01 trial	348	90.80%	33.91%	13.079	0.00	FAIL
2026-05-03 trial	281	100.00%	95.02%	11.138	91.46	PASS

The first clip is a useful failure case: it detects many frames but only 33.91% are usable, with a 50.29% high-jitter ratio. The second clip is the strong validation case: 95.02% usable pose coverage, zero missing frames, and 12/12 checks passing in both balanced and strict profiles.

Dual-Camera Timestamp-Synced Benchmark

May 16 Dual-Camera Diagnosis

Requested local pair:

• front: Recording from 2026-05-16 09-19-06.677315.mp4
• side/right: WhatsApp Video 2026-05-16 at 9.25.59 AM.mp4

The benchmark now pairs frames by timestamp instead of raw read order. With zero manual offset, the front video is sampled at about 29.997 FPS and the side/right video at about 20.546 FPS.

Backend/model	Frames	Front detect	Side/right detect	FPS	3D jitter	Mean reproj. error
MediaPipe lite	451	100.00%	93.35%	7.080	0.648368	7.974973 px
MediaPipe full	451	100.00%	100.00%	6.208	0.728271	8.184811 px
MediaPipe heavy	451	100.00%	100.00%	3.653	0.749542	7.163445 px
RTMPose-t	100	100.00%	100.00%	0.780	0.109600	7.565935 px
RTMPose-s	100	100.00%	100.00%	0.886	1.821319	9.344025 px
RTMPose-m	100	100.00%	100.00%	0.731	0.549703	9.698151 px

The RTMPose rows were run in a separate optional MMPose environment, not the default project venv. They use COCO-17 landmarks, while MediaPipe uses 33 pose landmarks, so the rows are valid evidence that the optional backend runs but are not direct anatomical superiority claims.

Manual offset sweep for MediaPipe full, 150 paired frames:

Right offset	Mean reproj. error	Triangulation	3D jitter
-1000 ms	7.354874 px	92.67%	0.671215
-750 ms	7.170307 px	93.33%	0.653630
-500 ms	7.785357 px	100.00%	0.617136
-250 ms	7.381756 px	100.00%	0.980109
0 ms	8.303597 px	100.00%	0.412678
250 ms	8.900566 px	100.00%	0.860295
500 ms	7.911324 px	100.00%	0.834469
750 ms	7.228909 px	99.33%	0.566801
1000 ms	6.554056 px	99.33%	0.655025

Best tested reprojection-error offset: +1000 ms. This improves the zero-offset mean reprojection error from 8.303597 px to 6.554056 px on the 150-frame sweep. A denser sweep around +1000 ms is the next calibration refinement.

Stage ablation on the same May 16 pair, MediaPipe full, 150 timestamp-synced frames:

Stage	Front detect	Side/right detect	Triangulation	Mean reproj. error	Issue
Detector only, no calibration	100.00%	100.00%	0.00%	not available	calibration intentionally disabled
Calibrated, zero offset	100.00%	100.00%	100.00%	8.597214 px	sparse segment pairs
Calibrated, best segment-aware offset +750 ms	offset sweep	offset sweep	99.33%	7.162890 px	sparse segment pairs

Limb visibility/depth-support rule: limb segments visible in both front and side/right views are counted as front-supported; side/right-only visible limbs are counted as side-only/back candidates. On the 150-frame MediaPipe full ablation, 56.67% of tracked limb segments were front-supported and 0.42% were side-only/back candidates. This is an occlusion diagnostic, not absolute depth ground truth.

Segment-length consistency columns are included (bone_length_cv_3d, mean_segment_length_drift, max_segment_length_drift, and per-segment CV for thigh, shank, upper arm, forearm, shoulder width, and pelvis width). On this specific run those values are null because triangulation retained sparse individual landmarks but no complete anatomical segment pairs with enough samples. Therefore bone-consistency improvement/worsening claims are not safe yet for this calibrated May 16 report.

Best current MediaPipe tradeoff on this pair:

• lite: fastest MediaPipe run, but missed side/right detections on 6.65% of paired frames.
• full: best balanced MediaPipe run with full detection in both views and stable bone-length CV of 0.094834.
• heavy: lowest mean reprojection error, but slowest MediaPipe run and no bone-length CV was available in this report.

These are self-consistency and calibrated-reconstruction diagnostics. They are not MPJPE or PA-MPJPE because no external ground-truth labels are present.

Earlier Weak-Front Pair

View	Frames	Detected frames	Detection rate	2D jitter	Avg confidence
Front	281	65	23.13%	0.104789	0.894668
Right	281	280	99.64%	0.040291	0.932542

Calibrated triangulation was available, but success was limited by the weaker front-view detection:

• triangulation success rate: 22.78%
• temporal 3D jitter: 0.358445
• 3D bone-length coefficient of variation: 0.208138
• benchmark throughput: 15.963 FPS

Validation Status

Claim	Evidence status	Safe to say?
Offline MediaPipe processing works	Supported by stable workflow	Yes
Dual-camera local API exists	Supported by backend docs	Yes
MMPose comparison supported	Optional path exists	Yes, but say optional
Better than MediaPipe	Not proven until benchmark filled	No
Accurate 3D pose	Not proven without ground truth	No
Calibrated 3D reconstruction	Only when valid calibration exists	Conditional
Research novelty	Pipeline/evaluation novelty	Yes, if phrased carefully

Recent Research & Reliability Enhancements

To transform the pipeline into a rigorous, defensible research tool, we have integrated:

• Explicit Stereo Calibration Validation: Automated structural verification of camera intrinsic matrices, finite values, and reprojection error thresholds prior to triangulation (src/stereo_calibration.py).
• Triangulation Reliability Diagnostics: Automatic categorization of 3D points into quality tiers (trusted, noisy, high_error, rejected) and tracking of reprojection error distributions (src/triangulation.py).
• Physics-Based Stabilization & Outlier Rejection: Kinematic sudden-jump filtering (max_jump_m=0.35m), anatomical bone-length hard rejection, and soft constraint smoothing while maintaining strict separation between raw observations and stabilized outputs (src/post_processor_3d.py).
• Real-Time Stability & Normative Baselines: Live bone-length coefficient of variation (CV) tracking with qualitative stability status reporting (src/realtime_stability.py), plus an automated normative human kinematics database across 5 standard movements (src/baseline_kinematics.py).
• Stage-by-Stage Latency Profiling: Standalone profiling tools (tools/profile_pipeline.py) identifying computational bottlenecks and theoretical maximum throughput across CPU and CUDA/TensorRT acceleration paths (docs/PERFORMANCE_AND_GPU_ARCHITECTURE.md).
• Trust Classification Engine: Automatically embeds a trust_report in results.json mapping calibration status, synchronization health, and reprojection error to one of four rigorous confidence levels (metric_3d_available, relative_3d_available, diagnostic_2d_only, or failed). This prevents bad data from silently leaking into exports.

Future Work

1. Ground Truth Validation: Integrate external optical marker ground-truth datasets (e.g., Human3.6M, TotalCapture) to compute formal MPJPE and PA-MPJPE error metrics.
2. Dense Multi-Camera Calibration: Extend automated sweep utilities to support >2 camera setups with bundle adjustment and dynamic extrinsic refinement.
3. Hardware Acceleration Expansion: Expand TensorRT and PyTorch CUDA batching to support real-time multi-person tracking at >60 FPS.

Novelty

This project does not introduce a new pose detector. Its contribution is a local-first motion-analysis and validation workbench around pretrained pose detectors, combining offline processing, dual-view reconstruction experiments, temporal quality metrics, kinematic exports, and reproducible baseline comparison.

Ground Truth Evaluation Plan

Current repository metrics evaluate detector coverage, temporal stability, missing landmarks, bone-length consistency, runtime, and calibrated reconstruction availability. Absolute 3D pose accuracy will require external ground-truth datasets or controlled measurement.

Planned datasets:

• Human3.6M for standard 3D pose metrics.
• TotalCapture for multi-view / Vicon-style validation.
• 3DPW for in-the-wild video testing.

Failure Cases

Failure case	Current behavior	Evidence
Poor front-view detection	paired triangulation succeeds only when both views detect pose	dual-camera benchmark
Optional MMPose not installed	comparison report records skipped backend rows	curated sample report
Missing/invalid calibration	2D metrics can run, calibrated 3D metrics are unavailable	benchmark smoke tests
No external ground truth	MPJPE/PA-MPJPE are not claimed	docs and validation status

Current Architecture

flowchart LR
  Video[Single video] --> Single[tools/process_video.py]
  Front[Front video] --> API[FastAPI upload API]
  Right[Right video] --> API
  API --> Queue[Local bounded job queue]
  Queue --> Dual[backend/pipeline.py]
  Single --> Metrics[Metrics + annotated video]
  Dual --> Sync[Frame sync]
  Sync --> Pose[MediaPipe pose]
  Pose --> Tri[Triangulation when calibrated]
  Tri --> Export[ExportManager]
  Metrics --> Runtime[.runtime/results]
  Export --> Runtime
  Runtime --> Viewer[Local viewer / reports]
  Video --> Compare[comparison experiments]
  LiveCam[Live webcam / IP camera] --> VS5[VS5 real-time GUI]
  VS5 --> CameraServer[CameraServer]
  CameraServer --> Master[MasterCoordinator]
  Master --> Live3D[Live 3D + realtime_stability]

Loading

The latest backend split is:

• backend/main.py: FastAPI endpoints, chunked uploads, job state, export downloads.
• backend/job_runner.py: one bounded local worker queue with startup recovery for queued jobs.
• backend/pipeline.py: orchestration for dual-view offline processing.
• backend/video_processing.py: video timestamps, frame reads, sync pairs, overlays.
• backend/pipeline_models.py: MediaPipe pose/face/hand landmarker factories; face and hand are optional.
• backend/triangulation_service.py: calibration loading and per-frame triangulation.
• backend/export_manager.py: stable JSON/CSV exports plus optional biomech formats when data and dependencies support them.

Limitations

• Ground truth: no marker/goniometer reference is committed. Reports use self-consistency metrics unless external labels are supplied.
• MMPose/RTMPose: optional dependency, not installed by default. This keeps the core install local and lightweight.
• MediaPipe models: .task files are local assets and ignored by Git. This avoids committing large vendor binaries.
• Live networking: restored for VS5 local experimentation, but still experimental compared with the offline/API reviewer path.
• Job durability: local bounded worker, not Celery/RQ. This is appropriate for workstation use.
• API security: local-first, not internet-hardened. Upload type/size validation and path redaction are now in place.
• 3D accuracy: requires valid calibration. Reports separate 2D detector metrics from calibrated 3D metrics.

One Clean Reproduction Path

Use one local video first. Keep generated files under .runtime/:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt

VIDEO=/absolute/path/to/video.mp4
python tools/process_video.py \
  --input "$VIDEO" \
  --output .runtime/results/demo_annotated.mp4 \
  --metrics-csv .runtime/results/demo_metrics.csv \
  --nodes-csv .runtime/results/demo_raw_3d_nodes.csv \
  --kinematics-csv .runtime/results/demo_kinematics.csv \
  --summary-json .runtime/results/demo_summary.json \
  --max-frames 300

Open the generated _raw_3d_nodes.csv with the local viewer:

python main_gui.py

Real-Time VS5 Mode

The restored live path is scoped to real-time orchestration. It does not replace the offline/comparison benchmark workflow.

# Single local camera, webcam, file, or stream URL
python launch_multi_camera.py --mode single --camera-source 0

# Remote camera/server laptop
python launch_multi_camera.py --mode server --camera-source 0

# Master laptop, connecting to the remote camera/server
python launch_multi_camera.py --mode master --remote-ip <SERVER_IP>

Live 3D output runs through timestamp-based frame synchronization, stereo triangulation with monocular/world-landmark fallback, 1-Euro 3D joint smoothing, live bone-length stabilization, and kinematics/dashboard diagnostics. The realtime_stability payload reports calibrated segments, pair-aware segment CV, mean/max segment drift, and corrected segments. These are live consistency diagnostics, not external ground-truth accuracy metrics.

Dual-Camera API

Run the local processing API:

uvicorn backend.main:app --host 127.0.0.1 --port 8000

The API supports:

• POST /upload: streamed front/right video upload, written in 1 MB chunks.
• POST /jobs/{job_id}/process: enqueue a local processing job.
• GET /jobs: list jobs.
• GET /jobs/{job_id}: inspect job state.
• GET /jobs/{job_id}/results: fetch completed results JSON.
• GET /jobs/{job_id}/exports: list generated export files.
• GET /jobs/{job_id}/exports/{export_key}: download a specific export.

The local worker is intentionally simple: one bounded background worker, no external broker, no distributed retry system. That is appropriate for this local workstation project and safer than creating one raw thread per request.

Upload guardrails:

• allowed video extensions: .mp4, .mov, .avi, .mkv, .webm
• allowed video content types for common video uploads
• streamed size limit via MOCAP_MAX_UPLOAD_BYTES
• job detail responses expose URLs/status flags instead of absolute local paths

Comparison And Ablation

Discover likely trial videos:

python comparison/experiments/discover_trial_videos.py --search-dir .

Run same-video MediaPipe baseline:

python comparison/experiments/compare_mediapipe_mmpose.py \
  --video /absolute/path/to/video.mp4 \
  --output comparison/reports/backend_comparison.json \
  --max-frames 300

Run optional MMPose/RTMPose comparison:

pip install -r requirements-mmpose.txt
python comparison/experiments/compare_mediapipe_mmpose.py \
  --video /absolute/path/to/video.mp4 \
  --include-mmpose \
  --mmpose-model human \
  --output comparison/reports/backend_comparison.json \
  --max-frames 300

Run dual-camera offline backend benchmark:

python comparison/experiments/benchmark_dualcam_offline.py \
  --front-video /absolute/path/front.mp4 \
  --right-video /absolute/path/right.mp4 \
  --calibration calibration_front_right.json \
  --backend both \
  --mmpose-model rtmpose-t,rtmpose-s \
  --device cpu \
  --max-frames 300 \
  --output-dir comparison/reports/dualcam_offline \
  --save-keypoints

Reports are written to comparison/reports/dualcam_offline/. See docs/DETAILED_COMPARISON_RESULTS.md for the results and ablation reporting format. Do not fill benchmark tables with invented values; use generated reports only.

MediaPipe-alone is a baseline, not ground truth. Ground truth requires an external reference such as labeled keypoints, marker-based capture, calibrated measurement targets, or goniometer angle measurements.

Status

Stable local paths:

• Offline single-video processing with annotated video, metrics CSV, raw node CSV, kinematics CSV, and summary JSON.
• Local dual-camera FastAPI processing jobs with chunked uploads and a bounded local worker queue.
• MediaPipe pose detection as the default backend.
• Optional face/hand dense landmark extraction when model files are present.
• Export service for JSON, raw 3D nodes, kinematics CSVs, and dense landmark schema. Biomech formats are optional and dependency/data-dependent.
• Local viewer/replay utilities for generated motion files.

Experimental paths:

• MMPose/RTMPose comparison backends.
• EasyMocap preserved comparison suite.
• Live multi-camera networking.
• Avatar retargeting quality across arbitrary rigs.

Removed paths:

• React/Vite frontend.
• Browser dashboard.
• FastAPI static frontend hosting.
• Node/npm deployment workflow.

Folder Map

backend/        Local FastAPI processing API and dual-view pipeline modules
src/            Core algorithms: detection, calibration, kinematics, exports
tools/          Local CLI workflows, validation tools, and viewer launchers
comparison/     MediaPipe/MMPose/EasyMocap benchmark sandbox
docs/           Current architecture, setup, workflow, and results documentation
tests/          Python tests
models/         Local model assets, ignored except README
.runtime/       Runtime uploads, outputs, results, logs, and reports

Runtime Files

Generated outputs belong under .runtime/, comparison/reports/, or another ignored runtime folder. Keep only small curated examples in Git, such as docs/results/sample_dual_view_result.json.

Quality Gate

CI runs the focused quality gate in .github/workflows/ci.yml. Run the same checks locally:

pytest \
  tests/test_extended_kinematics.py \
  tests/test_frame_quality.py \
  tests/test_known_angle.py \
  tests/test_known_distance.py \
  tests/test_static_jitter.py \
  tests/test_sync_calibration_validation.py \
  tests/test_comparison_video_discovery.py \
  tests/test_comparison_backends_contract.py \
  tests/test_dualcam_benchmark_smoke.py \
  tests/test_target_fps_sampling.py \
  tests/test_sync_sweep.py \
  tests/test_triangulation_reliability.py \
  tests/test_physics_stabilization.py \
  tests/test_baseline_kinematics.py \
  tests/test_realtime_stability.py \
  tests/test_validate_kinematics_and_3d.py

ruff check \
  backend/main.py backend/models.py backend/pipeline.py \
  backend/export_manager.py backend/job_runner.py backend/pipeline_models.py \
  backend/triangulation_service.py backend/video_processing.py \
  comparison/backends comparison/evaluation.py comparison/video_discovery.py \
  comparison/experiments/compare_mediapipe_mmpose.py \
  comparison/experiments/benchmark_dualcam_offline.py \
  comparison/experiments/discover_trial_videos.py \
  tests/test_comparison_video_discovery.py \
  tests/test_comparison_backends_contract.py \
  tests/test_dualcam_benchmark_smoke.py

black --check \
  backend/main.py backend/models.py backend/pipeline.py \
  backend/export_manager.py backend/job_runner.py backend/pipeline_models.py \
  backend/triangulation_service.py backend/video_processing.py \
  comparison/backends comparison/evaluation.py comparison/video_discovery.py \
  comparison/experiments/compare_mediapipe_mmpose.py \
  comparison/experiments/benchmark_dualcam_offline.py \
  comparison/experiments/discover_trial_videos.py \
  tests/test_comparison_video_discovery.py \
  tests/test_comparison_backends_contract.py \
  tests/test_dualcam_benchmark_smoke.py

mypy comparison/backends --ignore-missing-imports --follow-imports=skip
python tools/process_video.py --help
python comparison/experiments/benchmark_dualcam_offline.py --help