Skip to content

fix(review): compute public-stats accuracyPct from own-ledger merged/closed only - #7474

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
xfodev:fix/public-stats-accuracy-fold-7449
Jul 20, 2026
Merged

fix(review): compute public-stats accuracyPct from own-ledger merged/closed only#7474
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
xfodev:fix/public-stats-accuracy-fold-7449

Conversation

@xfodev

@xfodev xfodev commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #7449. getPublicStats computed the homepage's global totals.accuracyPct as accuracyPct(totals.merged, totals.closed, totals.reversed), but those inputs are drawn from structurally different, independently-growing populations: totals.merged/totals.closed are folded with the whole registered Orb fleet's counts (via getOrbGlobalStats, which has no reversal concept), while totals.reversed stays own-ledger-only. Since accuracyPct is 1 - min(1, reversed / (merged + closed)), the denominator grows with every newly registered install while the numerator stays fixed to the own-ledger repos — so the published accuracy trended toward 100% as more self-hosted installations registered, independent of the fleet's actual reversal behavior.

Fix (option 1 from the issue)

Snapshot the pre-fold own-ledger merged/closed (right beside the existing ownLedgerReviewed snapshot) and compute the global accuracyPct from those, so its numerator and denominator come from the same population. The Orb-fleet fold still (correctly) inflates the raw reviewed/handled/minutesSaved totals — those have no numerator/denominator pairing and are working as intended per the file's own comments. accuracyPct's formula/signature is untouched, and per-project byProject[].accuracyPct was already same-scope (each computed from its own merged/closed/reversed) and is unchanged. A code comment at the call site (and at the snapshot) documents the choice, matching the file's house style of documenting non-obvious metric-composition decisions.

Option 2 (fold a fleet-wide reversal count into totals.reversed) was not chosen: getOrbGlobalStats (src/orb/outcomes.ts) reports only { merged, closed, total } with no reversal tracking, so option 2 would require a schema/query extension for no additional accuracy over option 1.

Tests

test/unit/public-stats.test.ts adds a regression test running the real getPublicStats end-to-end (own-ledger disposition + reversal reads and the orb_pr_outcomes aggregate all mocked): 100 own-ledger decided PRs with 10 real reversals (a true 90% accuracy) plus a huge 10,000-PR Orb fleet fold. It asserts the raw counts still fold (merged/closed/handled) but accuracyPct is 90, not the pre-fix fleet-inflated 99, and that per-project accuracy stays 90. All existing getPublicStats field tests (reviewed/handled/minutesSaved/per-project accuracyPct, the Orb-fold tests) pass unchanged. Every changed line + branch is covered (verified via --coverage), so codecov/patch holds.

Validation

  • git diff --check
  • npm run typecheck (root tsc --noEmit) green
  • npm run test:coverage on the affected suite: 100% of changed lines + branches covered; existing public-stats unit + integration-route tests pass unmodified
  • Scope is src/review/public-stats.ts + its unit test only. The returned stats object's fields/shape are unchanged (same accuracyPct: number | null), so no OpenAPI/schema/generated artifact needs regeneration

Safety

  • No secrets, wallets, hotkeys/coldkeys, PATs, private keys, raw trust scores, private rankings, or maintainer evidence.
  • Public-facing metric change only makes the published number more honest (can no longer be inflated toward 100% by fleet growth); no new data is exposed.
  • Auth/CORS/session: N/A. API/OpenAPI/MCP: N/A (response shape unchanged). UI: N/A. No changelog edit; no site//CNAME/lovable changes.

Closes #7449

…closed only

getPublicStats computed the global totals.accuracyPct from the Orb-fleet-folded
totals.merged/closed (denominator) against the own-ledger-only totals.reversed
(numerator). getOrbGlobalStats has no reversal concept, so the denominator grew
with every newly registered install while the numerator stayed own-ledger-scoped,
trending the published accuracy toward 100% independent of real reversal behavior.

Snapshot the pre-fold own-ledger merged/closed (beside the existing ownLedgerReviewed
snapshot) and compute the global accuracyPct from those, so numerator and denominator
are drawn from the same population (option 1 of the issue). The fleet fold still
inflates reviewed/handled/minutesSaved, which have no numerator/denominator pairing;
accuracyPct's formula and byProject's per-project values are unchanged. Adds a
regression test asserting a huge Orb-fleet fold no longer pulls accuracyPct toward 100.

Closes JSONbored#7449
@xfodev
xfodev requested a review from JSONbored as a code owner July 20, 2026 11:30
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.35%. Comparing base (fbd1cba) to head (fdbe4ff).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7474      +/-   ##
==========================================
- Coverage   91.36%   91.35%   -0.02%     
==========================================
  Files         717      717              
  Lines       73016    73018       +2     
  Branches    21628    21628              
==========================================
- Hits        66713    66704       -9     
- Misses       5265     5272       +7     
- Partials     1038     1042       +4     
Flag Coverage Δ
shard-1 34.07% <100.00%> (-0.09%) ⬇️
shard-2 39.27% <0.00%> (-0.05%) ⬇️
shard-3 32.59% <0.00%> (-0.01%) ⬇️
shard-4 39.53% <100.00%> (+<0.01%) ⬆️
shard-5 36.57% <0.00%> (-0.02%) ⬇️
shard-6 34.32% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/review/public-stats.ts 97.46% <100.00%> (+0.06%) ⬆️

... and 2 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 20, 2026
@loopover-orb

loopover-orb Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-20 11:46:34 UTC

2 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This fixes the accuracyPct denominator-population mismatch identified in #7449 by snapshotting own-ledger merged/closed before the Orb fleet fold and computing accuracyPct from those pre-fold values instead of the fleet-inflated totals.merged/closed. The fix is correct: totals.reversed was already own-ledger-only, so pairing it with the pre-fold ownLedgerMerged/ownLedgerClosed restores same-population numerator/denominator, matching per-project accuracyPct's existing scope. The regression test constructs a realistic scenario (own-ledger 100 merged/10 reversed vs. a 10000-PR Orb fleet) and asserts both the correct 90% result and explicitly guards against the pre-fix 99% bug.

Nits — 3 non-blocking
  • src/review/public-stats.ts: ownLedgerMerged/ownLedgerClosed are read as totals.merged/totals.closed right before the Orb fold mutates those same fields in place — a very tight, easy-to-break invariant; consider renaming or adding a runtime assertion so a future reorder of these lines can't silently reintroduce fix(review): public-stats accuracyPct denominator includes Orb-fleet merged/closed but numerator (reversed) stays own-ledger-only #7449.
  • The file is now ~414 lines per the size-smell brief; the extensive header/inline commentary is valuable but consider whether some of the older Orb-fold rationale (already documented above) could be trimmed now that this PR adds more prose alongside it.
  • src/review/public-stats.ts:380-386: consider extracting the pre-fold snapshot values (ownLedgerReviewed, ownLedgerMerged, ownLedgerClosed) into a single small object to make the 'these three must be read before the Orb fold mutates totals' invariant more explicit and less prone to future edit-order bugs.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #7449
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 53 registered-repo PR(s), 29 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor xfodev; Gittensor profile; 53 PR(s), 0 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Addressed
The diff snapshots pre-fold own-ledger merged/closed and switches the global accuracyPct call site to use them alongside the own-ledger reversed count, matching option 1 from the issue, while leaving the Orb fleet fold intact for merged/closed/reviewed/handled/minutesSaved as required. A comment documents the choice at the call site and a regression test exercises getPublicStats end-to-end confirm

Review context
  • Author: xfodev
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: JavaScript, TypeScript
  • Official Gittensor activity: 53 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 14db2b8 into JSONbored:main Jul 20, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(review): public-stats accuracyPct denominator includes Orb-fleet merged/closed but numerator (reversed) stays own-ledger-only

1 participant