A real-time computer-vision coach for indoor climbing.
Point a camera at a bouldering wall and ClimbrerCoach tracks your body, finds the holds, and tells you — live, with an AR overlay and a voice — which limb to move to which hold next, why, and roughly how hard the route is.
| Capability | How it works |
|---|---|
| Pose tracking | MediaPipe Pose → 8 keypoints (wrists, ankles, hips, shoulders) |
| Hold detection | HSV color segmentation → hold centroids in the frame |
| Wall calibration | A 4-point homography maps pixels → real-world wall metres |
| Contact inference | Nearest-hold assignment per limb, smoothed with hysteresis to stop flicker |
| Balance model | Centre-of-mass vs. a Shapely support polygon → a signed stability margin |
| Move planning | greedy (instant), A*, and a real beam search over reachable holds |
| Top-k beta | The best move and ranked alternatives, drawn on the overlay |
| Grade estimation | An interpretable heuristic maps move-size / sparsity / balance / traverse → a V-grade |
| AI voice coach | Short spoken cues from the Claude API, with a deterministic offline fallback |
| Session reports | Every run logs to JSONL and renders heatmaps + an ascent timeline + a PDF |
Honesty note. The grade estimator is a documented heuristic (see
grade.py), not a trained model. The AI coach uses a real LLM when an API key is present and a rule-based generator otherwise — it never silently pretends. Where this README says "beam search" it means an actual beam search (search_beam.py), covered by tests.
flowchart LR
CAM[Camera / video] --> POSE[MediaPipe pose]
CAM --> HOLDS[HSV hold detection]
POSE --> HOMO[Homography px→m]
HOLDS --> HOMO
HOMO --> CONTACT[Contacts + hysteresis]
CONTACT --> STATE[COM + stability margin]
STATE --> PLAN{Planner<br/>greedy · A* · beam}
PLAN --> TOPK[Top-k suggestions]
PLAN --> GRADE[Grade estimator]
TOPK --> OVERLAY[AR overlay + HUD]
TOPK --> COACH[AI coach server]
COACH -->|Claude or rules| TTS[Voice cue]
OVERLAY --> LOG[(Session JSONL)]
GRADE --> LOG
LOG --> REPORT[Heatmaps · timeline · PDF]
The heavy lifting — ranking moves, A*/beam search, stability, grade estimation —
lives in camera-free, unit-tested modules under climbcoach_mvp/src/planner
and .../state. src/app.py is only the thin OpenCV/MediaPipe glue around them.
That separation is what lets the whole planning brain be tested in CI without a
webcam. See also docs/system_diagram.svg.
python -m venv .venv && source .venv/bin/activate
pip install -r climbcoach_mvp/requirements-dev.txt -r ai_coach_server/requirements-dev.txt
(cd climbcoach_mvp && pytest) # 40 tests: planner, search, stability, grade, sim
(cd ai_coach_server && pytest) # 6 tests: coach backends + endpointscd climbcoach_mvp
pip install -r requirements-dev.txt
python scripts/portfolio_demo.py
# -> runs/portfolio_report.pdf, heatmaps, and an ascent timelineWith no climbing video present, this drives a synthetic climber through the real
planning pipeline (src/sim/synthetic.py) — the log and grade are produced by
the actual code, not hand-authored. A sample of the output lives in
docs/demo_report.pdf.
cd climbcoach_mvp
pip install -r requirements.txt # installs OpenCV + MediaPipe
python src/app.py --source 0 --planner beam --topk 3 --ai_coach local
# press 'q' to quit; add --include_feet to rank foot moves tooCalibrate first for real-world coordinates (optional but recommended):
python scripts/set_homography.py --source 0 --width_m 4.0 --height_m 4.5
python scripts/calibrate_anthro.py --source 0 # capture your reach in a T-poseexport ANTHROPIC_API_KEY=sk-ant-... # optional — upgrades the coach to Claude
docker compose up --build- Coach API + docs: http://localhost:8000/docs (and
/healthshows the active backend) - Demo artifacts land in
climbcoach_mvp/runs/
Everything is scored in wall coordinates (metres); y = 0 is the top of the wall
(the goal), so "up" means a smaller y.
greedy—suggest.pyranks every reachable hold byheight + lateral + reach + instabilityand returns the best. Instant; the low-latency default. Also produces the top-k list.A*—search_astar.pyplans a full multi-move path to the top over climbing states(LH, RH, LF, RF).beam—search_beam.pykeeps the bestbeam_widthstates per depth for bounded lookahead.
A*/beam fall back to the greedy top-1 when a full plan isn't found in budget,
so a suggestion is always available. Reachability uses your calibrated arm/leg
span plus a dynamic-move factor; stability uses a Shapely support polygon.
difficulty = 4·reach_ratio + 2·sparsity + 2.5·instability + 1.5·lateral, binned
into VB…V10. Each term is interpretable and derived from the observed session —
see grade.py. It's a proxy, and it says so.
climbcoach_mvp/
src/
perception/ pose_tracker · holds_detector · homography (OpenCV/MediaPipe)
state/ contacts · contact_smoother · com · anthropometrics
planner/ suggest · greedy · search_astar · search_beam · plan_runner
cost_models · stability · reachability · grade · state_repr
sim/ synthetic — drives the real pipeline with no camera
coach/ overlay · ai_coach · ws_coach
app.py real-time glue (the only camera-coupled file)
scripts/ portfolio_demo · generate_demo · replay_heatmap · report_portfolio
set_homography · calibrate_anthro
tests/ 40 tests, all camera-free
ai_coach_server/ FastAPI service: Claude backend + offline rule-based fallback
docs/ system diagram + sample PDF report
.github/workflows/ci.yml pytest on Python 3.11 / 3.12 / 3.13
CI runs both test suites on Python 3.11–3.13 (.github/workflows/ci.yml). The
core tests deliberately avoid importing OpenCV/MediaPipe, so the planning brain is
verified quickly and reproducibly — including an end-to-end synthetic climb that
must ascend and produce a coherent grade.
- Multi-color hold detection (learn several HSV bands, or a lightweight detector)
- A trained grade model to complement the heuristic
- Record a real climbing clip and ship a rendered demo GIF
- Per-move feedback replay in the PDF report