Add snooker private challenge v0

documentation/snooker-private-challenge-v0-brief.md

@@ -0,0 +1,112 @@
+# Snooker Private Challenge V0 Brief
+
+## Decision
+
+Use the existing TurboVision private-track architecture and add a narrow snooker element:
+
+```text
+manako/DetectSnookerBallState
+```
+
+This is not a redesign. It is the smallest rigorous path to a live snooker CV challenge: validators send a short clip plus explicit target frame ids, miners return ball states for those frames, and scoring stays inside the existing private runner/weights/audit flow.
+
+## What This Branch Provides
+
+- Public request contract: `video_url` plus `target_frames`
+- Miner response contract: `prediction.type = "snooker_ball_state"` with `frames[].balls[]`
+- Snooker manifest support: `groundtruth_type = "snooker_ball_state"` and `metrics.pillars.snooker_ball_state`
+- Parser support for backend `/api/challenge/v3` snooker tasks
+- Miner forwarding of `target_frames`
+- Runner scoring/upload support for snooker
+- Spotcheck rescoring support for snooker
+- Score-scaled private weighting for snooker, matching cricket behavior
+- Hardened ball-state scorer with target-frame-only evaluation and red Hungarian matching
+- Miner-facing spec and backend/data handoff docs
+- Full local validator/private test coverage
+
+## V0 Challenge Shape
+
+The backend returns a normalized 20 second clip and fixed target frames:
+
+```json
+{
+  "task_id": "12345",
+  "payload": {
+    "clip_url": "https://assets.example.com/snooker/window-abc.mp4",
+    "target_frames": [50, 150, 250, 350, 450]
+  }
+}
+```
+
+The miner returns table-normalized ball states:
+
+```json
+{
+  "challenge_id": "12345",
+  "prediction": {
+    "type": "snooker_ball_state",
+    "frames": [
+      {
+        "frame": 50,
+        "balls": [
+          {"label": "cue", "x": 0.52, "y": 0.33, "state": "on_table"},
+          {"label": "black", "state": "occluded"}
+        ]
+      }
+    ]
+  },
+  "processing_time": 0.82
+}
+```
+
+## Scoring Summary
+
+Only requested target frames are scored. Missing requested frames score zero; extra non-target frames are ignored.
+
+The scorer rewards coordinate accuracy, identity accuracy, red count accuracy, state accuracy, and low false-positive rate. It penalizes duplicate unique colours, wrong red counts, missing balls, extra balls, invalid labels, empty predictions, and `on_table` balls without coordinates.
+
+Canonical labels are `cue`, `red`, `yellow`, `green`, `brown`, `blue`, `pink`, `black`. Reds are matched as an unordered set using Hungarian assignment.
+
+Canonical states are `on_table`, `potted`, `occluded`, `unknown`.
+
+## What Is Still External
+
+The branch deliberately does not implement the external backend or publish the live manifest. The remaining launch work is:
+
+- create 6 public bootstrap windows
+- create 20 hidden scoring windows
+- normalize clips to 20 seconds, 25 fps, max 480p
+- annotate hidden ground truth for frames `[50, 150, 250, 350, 450]`
+- host private clips and keep hidden source ids/timestamps unpublished
+- implement backend `/api/challenge/v3` and `/api/tasks/{id}/ground-truth` responses using the branch contract
+- publish the manifest element after backend/data are ready
+
+## Manifest Snippet
+
+```yaml
+id: manako/DetectSnookerBallState
+track: private
+groundtruth_type: snooker_ball_state
+weight: 0.05
+window_block: 300
+eval_window: 4
+metrics:
+  pillars:
+    snooker_ball_state: 1.0
+targets:
+  snooker_ball_state: 0.85
+baselines:
+  snooker_ball_state: 0.0
+first_block: <current_publish_block + 100>
+```
+
+## Verification
+
+Current branch verification:
+
+```text
+python -m compileall -q scorevision tests
+python -m pytest tests/private tests/validator
+```
+
+Result: `155 passed`.

documentation/snooker-private-challenge-v0.md

@@ -0,0 +1,114 @@
+# Snooker Private Challenge V0
+
+This branch defines the TurboVision-side contract for `manako/DetectSnookerBallState`. It does not implement the external challenge backend or publish the live manifest.
+
+## Challenge Shape
+
+Validators request one short video window and a fixed frame set:
+
+```json
+{
+  "task_id": "12345",
+  "payload": {
+    "clip_url": "https://assets.example.com/snooker/window-abc.mp4",
+    "target_frames": [50, 150, 250, 350, 450]
+  }
+}
+```
+
+`video_url` may be provided at top level instead of `payload.clip_url`. For snooker, a video URL and non-empty `target_frames` are required. Frame ids are relative to the supplied 20 second clip.
+
+## Ground Truth Shape
+
+`GET /api/tasks/{id}/ground-truth` must return:
+
+```json
+{
+  "ground_truth": {
+    "frames": [
+      {
+        "frame": 50,
+        "balls": [
+          {
+            "label": "cue",
+            "x": 0.52,
+            "y": 0.33,
+            "state": "on_table",
+            "confidence": 1.0,
+            "bbox": [120, 90, 135, 105]
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+Hidden annotation storage should also retain table corners and orientation for QA. V0 miner responses do not require table-corner output.
+
+## Data Target
+
+Prototype dataset target:
+
+- 6 public bootstrap windows
+- 20 hidden/private scoring windows
+- Candidate source footage may be YouTube prototype footage for dry run
+- Hidden clip ids, source timestamps, and ground truth must not be published
+
+Normalize every scoring clip before annotation:
+
+- 20 seconds
+- 25 fps
+- max height 480p
+- target frames `[50, 150, 250, 350, 450]`
+
+## Manifest Snippet
+
+Use this as the private manifest element once the backend and hidden data are ready:
+
+```yaml
+id: manako/DetectSnookerBallState
+track: private
+groundtruth_type: snooker_ball_state
+weight: 0.05
+window_block: 300
+eval_window: 4
+metrics:
+  pillars:
+    snooker_ball_state: 1.0
+targets:
+  snooker_ball_state: 0.85
+baselines:
+  snooker_ball_state: 0.0
+first_block: <current_publish_block + 100>
+```
+
+## V0 Scoring Contract
+
+The validator scores only requested `target_frames`. Missing requested frames score zero for that frame, and extra non-target frames are ignored. Duplicate response objects for a requested target frame are treated as contract violations: their ball lists are merged for scoring and the final frame-set score is penalized.
+
+Labels are canonical: `cue`, `red`, `yellow`, `green`, `brown`, `blue`, `pink`, `black`. Unique colours match by label. Reds match as an unordered set using Hungarian assignment in normalized table coordinates.
+
+States are canonical: `on_table`, `potted`, `occluded`, `unknown`. Invalid state strings are not normalized to `unknown`; they are scored as invalid predictions. `on_table` balls require `x` and `y`; `potted`, `occluded`, and `unknown` may omit coordinates.
+
+On-table coordinate scoring uses normalized table distance with a `0.05` tolerance. A perfect coordinate match receives full coordinate credit; coordinate credit decays linearly to zero at or beyond the tolerance. On-table predictions outside that tolerance are not counted as matched balls. Red-count accuracy only counts valid red predictions, so invalid red entries cannot earn red-count credit.
+
+Breakdown keys remain:
+
+- `coordinate_accuracy`
+- `identity_accuracy`
+- `red_count_accuracy`
+- `state_accuracy`
+- `false_positive_score`
+- `snooker_ball_state`
+
+## External Dependencies
+
+Backend work still required:
+
+- `/api/challenge/v3` must emit snooker tasks with a clip URL and target frames
+- `/api/tasks/{id}/ground-truth` must emit `ground_truth.frames`
+- private data hosting must provide short normalized clips
+- hidden annotation and QA must be produced outside this repo
+
+Rights/leakage note: public YouTube footage is acceptable only for dry-run examples. Hidden scoring should avoid exposed timestamps and ideally use rights-cleared or non-public assets.

scorevision/__init__.py

@@ -1,20 +1,23 @@
 import asyncio
 from asyncio import run
+import importlib
 from logging import getLogger
 from pathlib import Path
 import click
-from scorevision.cli.audit_validator import audit_validator
-from scorevision.cli.benchmark import benchmark_cli
-from scorevision.cli.central_validator import central_validator
-from scorevision.cli.elements import elements_cli
-from scorevision.cli.index_maintenance import index_cli
-from scorevision.cli.manifest import manifest_cli
 from scorevision.utils.logging import setup_logging
 from scorevision.utils.settings import get_settings
 
 logger = getLogger(__name__)
 
 
+def _add_optional_command(module_name: str, command_name: str) -> None:
+    try:
+        module = importlib.import_module(module_name)
+        app.add_command(getattr(module, command_name))
+    except Exception as e:
+        logger.debug("Optional CLI command %s.%s unavailable: %s", module_name, command_name, e)
+
+
 @click.group(name="sv")
 @click.option(
     "-v",
@@ -126,9 +129,9 @@ def validate_cmd(tail: int, m_min: int, tempo: int, manifest_path):
     )
 
 
-app.add_command(audit_validator)
-app.add_command(benchmark_cli)
-app.add_command(central_validator)
-app.add_command(index_cli)
-app.add_command(manifest_cli)
-app.add_command(elements_cli)
+_add_optional_command("scorevision.cli.audit_validator", "audit_validator")
+_add_optional_command("scorevision.cli.benchmark", "benchmark_cli")
+_add_optional_command("scorevision.cli.central_validator", "central_validator")
+_add_optional_command("scorevision.cli.index_maintenance", "index_cli")
+_add_optional_command("scorevision.cli.manifest", "manifest_cli")
+_add_optional_command("scorevision.cli.elements", "elements_cli")

scorevision/cli/benchmark.py

@@ -3,11 +3,9 @@
 from json import dumps, loads
 from pathlib import Path
 
-import bittensor as bt
 import click
 
 from scorevision.miner.open_source.chute_template.schemas import TVPredictInput
-from scorevision.utils.bittensor_helpers import get_subtensor
 from scorevision.utils.cloudflare_helpers import get_s3_client
 from scorevision.utils.manifest import get_current_manifest, load_manifest_from_public_index
 from scorevision.utils.miner_registry import get_miners_from_registry
@@ -134,6 +132,8 @@ async def _resolve_private_target(
     element_id: str,
     winner_entry: dict,
 ):
+    from scorevision.utils.bittensor_helpers import get_subtensor
+
     settings = get_settings()
     winner_hotkey = str(winner_entry.get("winner_hotkey") or "").strip()
     if not winner_hotkey:
@@ -186,6 +186,8 @@ async def run_top_performer_benchmark(
     success_count = 0
 
     if track == "private":
+        import bittensor as bt
+
         target_miner = await _resolve_private_target(selected_element_id, winner_entry)
         wallet = bt.wallet(
             name=settings.BITTENSOR_WALLET_COLD,

scorevision/miner/private_track/MINER.md

@@ -192,11 +192,12 @@ Current private elements are served as separate tracks from the miner point of v
 
 - Football private element (`groundtruth_type=soccer_action`)
 - Cricket private element (`groundtruth_type=cricket_delivery`)
+- Snooker ball-state private element (`groundtruth_type=snooker_ball_state`)
 
 The template miner uses a static mode switch in `scorevision/miner/private_track/routes.py`:
 
 ```python
-MINER_MODE = "soccer_action"  # or "cricket_delivery"
+MINER_MODE = "soccer_action"  # or "cricket_delivery" or "snooker_ball_state"
 ```
 
 Keep one deployment per mode. Do not expect one running container to handle both private element types automatically.
@@ -238,6 +239,7 @@ Keep one deployment per mode. Do not expect one running container to handle both
 ```
 
 For full cricket field guidance, see `scorevision/miner/CRICKET_MINER_SPEC.md`.
+For snooker ball-state guidance, see `scorevision/miner/private_track/SNOOKER_MINER_SPEC.md`.
 
 ### 1.3 On-Chain Commitment Must Match Element
 
@@ -255,7 +257,7 @@ Example commitment shape (conceptual):
 }
 ```
 
-If you serve cricket, use the cricket element id instead.
+If you serve cricket or snooker, use that element id instead.
 
 ### 2. Test Locally (Without Wallet)
 
@@ -318,12 +320,12 @@ The CLI will:
 
 After deployment, **share with Score** (see [GHCR Setup](#ghcr-setup) step 4).
 
-## Private Miner Checklist (Football / Cricket)
+## Private Miner Checklist (Football / Cricket / Snooker)
 
 Before declaring your miner live:
 
 1. Set `MINER_MODE` to the intended private element type.
-2. Ensure predictor output matches the expected response payload (`soccer_action.items` or `cricket_delivery.item`).
+2. Ensure predictor output matches the expected response payload (`soccer_action.items`, `cricket_delivery.item`, or `snooker_ball_state.frames`).
 3. Deploy image and commit with the correct private `element_id`.
 4. Verify your axon IP/port is reachable.
 5. Confirm Score has GHCR read access to your package.