Single-player tracking pipeline that fuses YOLO detection, BoT-SORT multi-object tracking, ReID appearance embeddings, jersey number OCR, and full-body pose estimation into a real-time annotated video analysis tool — served through a Gradio web interface.
- Overview
- Architecture
- Pipeline Flow
- Module Reference
- Confidence Scoring System
- ReID Recovery Flow
- Installation
- Usage
- Configuration
- Project Structure
- Tech Stack
Vision solves a specific challenge in sports analytics: reliably tracking one player across an entire video clip, even when they are occluded, leave the frame, or look visually similar to teammates.
The pipeline operates in two stages:
| Stage | Purpose | Models Used |
|---|---|---|
| Initial Lock | Identify the target player on frame 0 | Florence-2 VLM (text prompt) or click-to-select |
| Continuous Track | Follow the target through the whole video | YOLO11 + BoT-SORT + ReID + OCR + Pose |
Once locked, the system never calls the VLM again. All future recovery uses lightweight ReID embeddings (~1 ms vs ~500 ms), keeping the pipeline real-time capable.
graph TD
subgraph Input
V[📹 Video Upload]
T[💬 Text Prompt]
C[🖱️ Click to Select]
end
subgraph Frame0["Frame 0 — Target Locking"]
YD[YOLO Detector<br/>detect only]
FL[Florence-2 VLM<br/>Caption → Phrase Grounding]
CS[Click-Select<br/>Closest Bbox Match]
IOU[IoU Matching<br/>VLM ↔ YOLO]
TID[Target Tracker ID<br/>Assigned by BoT-SORT]
REF[Reference Embedding<br/>HSV Color Histogram]
end
subgraph Loop["Per-Frame Processing Loop"]
DT[YOLO11 + BoT-SORT<br/>Detection + Tracking]
OCR[Jersey OCR<br/>EasyOCR every 15 frames]
POSE[Pose Estimator<br/>YOLOv8n-pose every N frames]
CONF[Confidence Scorer<br/>Detection · Tracker · OCR · ReID]
REID[ReID Recovery<br/>Cosine Similarity Relock]
VIZ[Visualizer<br/>BBox · Trace · Heatmap · Skeleton]
STAT[Stats Collector<br/>Centroid · Speed · Time]
end
subgraph Output
OV[📼 Annotated Video]
AN[📊 Analytics JSON]
PL[📈 Confidence Plot]
LOG[📋 Processing Log]
end
V --> YD
T --> FL
C --> CS
YD --> FL
YD --> CS
FL --> IOU
IOU --> TID
CS --> TID
TID --> REF
REF --> Loop
Loop --> DT
DT --> OCR
DT --> POSE
DT --> CONF
CONF -->|"score < 0.3 for 10 frames"| REID
REID -->|new tracker_id| DT
DT --> VIZ
DT --> STAT
Loop --> OV
Loop --> AN
Loop --> PL
Loop --> LOG
sequenceDiagram
participant UI as Gradio UI
participant APP as app.py
participant DT as YOLODetectorTracker
participant TL as TargetLocker (Florence-2)
participant CONF as ConfidenceScorer
participant REID as ReIDRecovery
participant VIZ as Visualizer
UI->>APP: process_video(video, prompt)
note over APP: Phase 1 — Frame 0 Setup
APP->>DT: detect(frame_0)
DT-->>APP: detections (no tracker_id)
APP->>DT: track_first_frame(frame_0)
DT-->>APP: detections with tracker_id
alt Text prompt
APP->>TL: lock_target(frame_0, prompt, detections)
TL-->>APP: target_id via VLM + IoU match
else Click-to-select
APP->>APP: _click_select_target(x, y, detections)
APP-->>APP: target_id via nearest bbox
end
APP->>DT: get_reid_embedding(frame_0, target_bbox)
DT-->>APP: reference_embedding (HSV histogram)
APP->>CONF: set_reference(reid_embedding)
APP->>REID: set_reference(embedding, bbox)
note over APP: Phase 2 — Processing Loop (all frames)
loop For each frame
APP->>DT: track(frame)
DT-->>APP: sv.Detections with tracker_id
opt every 15 frames
APP->>APP: jersey_ocr.detect_number(frame, bbox)
end
opt every N frames
APP->>APP: pose_estimator.estimate(frame)
end
APP->>CONF: score(frame_idx, detections, target_id, jersey, embedding)
CONF-->>APP: FrameScore (composite, detection_conf, reid_similarity)
alt confidence.needs_recovery()
APP->>REID: attempt_recovery(frame, detections, target_id)
REID-->>APP: new_target_id (or None)
end
APP->>VIZ: annotate(frame, detections, target_id, jersey, confidence)
VIZ-->>APP: annotated frame
end
APP->>UI: (output_video, stats_json, log, confidence_timeline)
classDiagram
class YOLODetectorTracker {
+model: YOLO
+conf: float
+classes: list
+tracker_yaml: str
+detect(frame) Detections
+track(frame) Detections
+track_first_frame(frame) Detections
+get_reid_embedding(frame, bbox) ndarray
+_compute_appearance_embedding(crop) ndarray$
+reset()
}
class TargetLocker {
-_model: Florence2
-_processor: AutoProcessor
+lock_target(frame, prompt, detections, tracked) int
-_load_model()
}
class ConfidenceScorer {
+WEIGHT_DETECTION: 0.30
+WEIGHT_TRACKER: 0.25
+WEIGHT_OCR: 0.15
+WEIGHT_REID: 0.30
+history: List~FrameScore~
+set_reference(jersey, reid_embedding)
+score(frame_idx, detections, target_id, ...) FrameScore
+needs_recovery() bool
+get_summary() dict
+get_confidence_timeline() dict
}
class ReIDRecovery {
-_reference_embedding: ndarray
-_similarity_threshold: float
-_recovery_count: int
+set_reference(embedding, bbox)
+attempt_recovery(frame, detections, target_id) int
+update_reference(frame, bbox, blend_factor)
}
class JerseyOCR {
-_reader: EasyOCR
-_cache: Dict~int, str~
+detect_number(frame, bbox) str
}
class PoseEstimator {
+estimate(frame) PoseResult
+get_target_pose(result, bbox) ndarray
}
class Visualizer {
+annotate(frame, detections, target_id, jersey, confidence) ndarray
}
class StatsCollector {
+frames_visible: int
+centroids: List
+update(detections, target_id)
+compute() dict
}
class FrameScore {
+frame_idx: int
+detection_conf: float
+tracker_quality: float
+ocr_consistency: float
+reid_similarity: float
+composite: float
+target_visible: bool
}
YOLODetectorTracker --> TargetLocker : provides detections for
YOLODetectorTracker --> ReIDRecovery : embedding function
ConfidenceScorer --> FrameScore : produces
ConfidenceScorer --> ReIDRecovery : triggers when score < threshold
Visualizer --> StatsCollector : reads centroids
Each frame produces a FrameScore — a weighted composite of four independent signals:
flowchart LR
subgraph Signals
D["🎯 Detection Confidence<br/><i>YOLO bbox score</i><br/>weight: 30%"]
TR["🏃 Tracker Quality<br/><i>Track age + streak</i><br/>weight: 25%"]
OCR["🔢 OCR Consistency<br/><i>Jersey # match</i><br/>weight: 15%"]
R["👤 ReID Similarity<br/><i>Cosine distance</i><br/>weight: 30%"]
end
subgraph Composite
C["⚡ Composite Score<br/>[0.0 → 1.0]"]
end
subgraph Decision
OK["✅ Continue tracking"]
LOW["⚠️ Low confidence<br/>counter + 1"]
REC["🔄 Trigger ReID Recovery<br/>10 consecutive low frames"]
end
D --> C
TR --> C
OCR --> C
R --> C
C -->|"score ≥ 0.3"| OK
C -->|"score < 0.3"| LOW
LOW -->|"10 frames"| REC
| Signal | Source | Description |
|---|---|---|
detection_conf |
YOLO | Raw bbox confidence score from the detector |
tracker_quality |
BoT-SORT | Track age normalized + consecutive match streak |
ocr_consistency |
EasyOCR | 1.0 if jersey number matches expected, 0.0 otherwise |
reid_similarity |
HSV Histogram | Cosine similarity between current crop and reference embedding |
When the tracker loses confidence, the system attempts to re-lock the correct player using only appearance — no VLM re-grounding required:
flowchart TD
A[Confidence drops below 0.3] --> B{Consecutive<br/>low frames ≥ 10?}
B -- No --> C[Keep current target_id]
B -- Yes --> D[Extract appearance embedding<br/>for every visible detection]
D --> E[Compute cosine similarity<br/>against reference embedding]
E --> F{Any similarity<br/>≥ 0.6 threshold?}
F -- No --> G[Target considered lost<br/>recovery_count++]
F -- Yes --> H[Switch to best-match tracker_id]
H --> I[Update reference embedding<br/>via EMA blend: α=0.1]
I --> J[Resume tracking with new ID]
G --> K[Continue loop — may recover<br/>on future frames]
Why ReID instead of re-running Florence-2?
| Method | Latency | GPU cost | Works mid-clip |
|---|---|---|---|
| Florence-2 re-grounding | ~500 ms | High | Requires clean frame |
| ReID cosine match | ~1 ms | Minimal | Any frame |
- Python 3.10+
- CUDA 11.8+ (optional, falls back to CPU)
- 4 GB RAM minimum (8 GB recommended for GPU inference)
# Clone the repository
git clone https://github.com/your-org/vision.git
cd vision/model
# Install dependencies
pip install -r requirements.txt
# Verify YOLO model is present
ls yolo11n.ptrequirements.txt installs:
ultralytics>=8.2.0 # YOLO11 + BoT-SORT tracker
supervision>=0.21.0 # Detection utilities & annotators
gradio>=4.0.0 # Web UI
opencv-python>=4.8.0 # Video I/O
torch>=2.0.0 # Deep learning backend
transformers>=4.40.0 # Florence-2 VLM
easyocr>=1.7.0 # Jersey number OCR
matplotlib>=3.7.0 # Confidence timeline plots
einops, timm, Pillow, numpy
Note: Florence-2 weights are downloaded automatically from HuggingFace on first run (~500 MB). Set
HF_HOMEto control the cache directory.
cd model
python app.pyThe UI is served at http://0.0.0.0:7860 by default.
- Upload an
.mp4video file - Select Text Prompt mode
- Enter a description:
"player wearing red jersey number 8" - Click Process Video →
- Upload the video — the first frame appears automatically
- Switch to Click to Select mode
- Click directly on the player you want to track
- Click Process Video →
| Output | Description |
|---|---|
| Tracked Video | Annotated .mp4 with bounding box, trace, heatmap, jersey number, and confidence indicator |
| Stats JSON | Time on screen, distance traveled, speed, jersey number, ReID recovery count |
| Confidence Plot | Per-frame composite / detection / ReID score timeline |
| Processing Log | Full console output for debugging |
All tunable parameters live in config.py:
# ── Detection ─────────────────────────────────────────────────────────
YOLO_MODEL = "yolo11n.pt" # Swap to yolo11s.pt for higher accuracy
YOLO_CONF_THRESHOLD = 0.25
YOLO_CLASSES = [0, 32] # COCO: 0=person, 32=sports ball
# ── Performance ───────────────────────────────────────────────────────
USE_FP16 = True # Half-precision on CUDA
FRAME_SKIP_INTERVAL = 2 # Detect every N frames
# ── OCR ───────────────────────────────────────────────────────────────
OCR_RUN_INTERVAL = 15 # Run OCR every N frames
OCR_CONFIDENCE_THRESHOLD= 0.3
TORSO_CROP_RATIO = 0.5 # Upper 50% of bbox for torso
# ── Pose ──────────────────────────────────────────────────────────────
POSE_ENABLED = True
POSE_MODEL = "yolov8n-pose.pt"
POSE_RUN_INTERVAL = 1
# ── Confidence & Recovery ─────────────────────────────────────────────
CONFIDENCE_RECOVERY_THRESHOLD = 0.3 # Score below this → low confidence
CONFIDENCE_RECOVERY_INTERVAL = 10 # N consecutive low frames → ReID trigger
REID_SIMILARITY_THRESHOLD = 0.6 # Cosine similarity required for re-lock
REID_EMBEDDING_UPDATE_RATE = 0.1 # EMA blend factor for reference update
# ── Visualization ─────────────────────────────────────────────────────
TRACE_LENGTH = 100 # Past positions drawn by TraceAnnotator
HEATMAP_RADIUS = 40| Scenario | Recommended change |
|---|---|
| Slow CPU | Increase FRAME_SKIP_INTERVAL to 4–6 |
| High occlusion | Lower REID_SIMILARITY_THRESHOLD to 0.45 |
| Crowded court | Increase YOLO_CONF_THRESHOLD to 0.4 |
| Better accuracy | Switch to yolo11s.pt or yolo11m.pt |
model/
├── app.py # Gradio UI + end-to-end pipeline orchestration
├── config.py # All tunable parameters
├── botsort.yaml # BoT-SORT tracker configuration
├── yolo11n.pt # YOLO11 nano model weights
├── requirements.txt
│
├── core/
│ ├── __init__.py
│ ├── detector.py # YOLODetector — detection-only wrapper
│ ├── detector_tracker.py # YOLODetectorTracker — YOLO + BoT-SORT unified
│ ├── target_locker.py # Florence-2 VLM target identification (frame 0)
│ ├── reid_recovery.py # Appearance-based re-locking (all future frames)
│ ├── confidence_scorer.py # Per-frame composite score (4 signals)
│ ├── jersey_ocr.py # EasyOCR jersey number detection
│ ├── pose_estimator.py # YOLOv8-pose estimation + biomechanics helpers
│ ├── pose_stats.py # Aggregate pose statistics per video
│ ├── visualizer.py # Supervision annotators (bbox, trace, heatmap, pose)
│ ├── stats.py # StatsCollector — centroid, speed, time
│ └── video_io.py # Frame extraction, generator, writer utilities
│
├── tests/
│ ├── __init__.py
│ ├── test_pipeline.py # Integration tests for the full pipeline
│ └── test_target_locker.py # Unit tests for IoU matching and VLM output parsing
│
├── output/ # Generated annotated videos (git-ignored)
│
└── Setup Gradio UI Project/ # React + Tailwind frontend scaffold (alternative UI)
├── src/
│ ├── app/App.tsx
│ └── components/
│ └── dashboard/
│ ├── AnalyticsPanel.tsx
│ ├── VideoPanel.tsx
│ └── HeaderBar.tsx
├── package.json
└── vite.config.ts
| Layer | Technology | Role |
|---|---|---|
| Detection | YOLO11n (ultralytics) |
Person + ball detection, 25 ms/frame on CPU |
| Tracking | BoT-SORT with ReID (built into Ultralytics) | Stable ID assignment across frames |
| Target lock | Florence-2-base (transformers) |
One-time text-to-bbox grounding on frame 0 |
| Re-identification | HSV histogram + cosine similarity | Lightweight appearance embeddings for recovery |
| OCR | EasyOCR | Digit-only jersey number detection |
| Pose | YOLOv8n-pose (ultralytics) |
17-keypoint COCO body estimation |
| Annotations | Supervision | BBox, trace, heatmap, label annotators |
| UI | Gradio 4.x | Web interface with progress streaming |
| Video I/O | OpenCV (cv2) |
Frame extraction and video writing |
| Inference backend | PyTorch (CUDA or CPU) | FP16 acceleration when GPU available |
The ReID system doesn't use a pretrained embedding network — it computes a fast, deterministic feature vector from each player crop:
Player crop (64×128 px)
│
▼
Convert to HSV
│
┌────┴──────────────────────────────────┐
│ │
▼ ▼
Hue histogram (32 bins) Spatial split (upper / lower half)
Sat histogram (32 bins) Upper hue histogram (16 bins) → torso color
Val histogram (32 bins) Lower hue histogram (16 bins) → shorts color
│ │
└────────────────┬──────────────────────┘
▼
Concatenate → 128-dim vector
│
▼
L2 normalize
│
▼
Reference embedding stored at lock time
EMA-updated (α=0.1) every frame the target is visible
This approach deliberately encodes jersey color and shorts color separately, making it robust to partial occlusion — a player with half their torso hidden still matches via their shorts histogram.
- Florence-2 load time — first run downloads ~500 MB of weights and takes 10–30 s to initialize. Subsequent runs reuse the cached model.
- ReID embedding quality — the HSV histogram embedding is lightweight but not rotation-invariant. Players viewed from very different angles may not match across camera cuts.
- OCR accuracy — jersey numbers smaller than ~40 px in the original frame often fail EasyOCR detection. Increasing
TORSO_CROP_RATIOcan help for larger players. - Multi-camera — the tracker resets between videos. There is no cross-clip identity persistence.
- Ball tracking — COCO class 32 (sports ball) is detected but not separately tracked or analyzed in the current stats output.
MIT — see LICENSE for details.
Built with YOLO11 · Florence-2 · BoT-SORT · EasyOCR · Supervision · Gradio