feat(observability): emit review quality to PostHog as $ai_metric - #10228
Conversation
Reviewer stances and inter-run agreement already reach SQL, Grafana and the maintainer recap, but never PostHog -- so cost, model and latency were readable there while whether the review was any GOOD was not, and the two halves could not be joined. Emit them as $ai_metric against the review's own trace, so quality lands beside the spend and the model that produced it. The property contract is taken verbatim from @posthog/core's captureTraceMetric -- name, value, trace id, with the value stringified as the SDK does it -- so a hand-built metric and an SDK-built one are indistinguishable downstream. The trace id defaults to the ambient OTel trace every generation under the review already carries. No ambient trace means no event: an orphan quality score joins to nothing and would only inflate counts. An uncorroborated review reports NO agreement rather than the UNCORROBORATED_AGREEMENT floor. That constant is a deliberate placeholder, not a measurement, and publishing it would drop a fabricated 0.5 into an average alongside real scores. The decision lives in a pure helper beside the scorer, so both sides of it are testable without driving a whole dual-reviewer run. Closes #10226
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-31 13:38:01 UTC
Review summary Nits — 2 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionPartially addressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10228 +/- ##
==========================================
- Coverage 92.22% 91.35% -0.87%
==========================================
Files 937 937
Lines 114376 114389 +13
Branches 27654 27658 +4
==========================================
- Hits 105482 104501 -981
- Misses 7588 8777 +1189
+ Partials 1306 1111 -195
Flags with carried forward coverage won't be shown. Click here to find out more.
|
1 participant
Summary
Reviewer stances and inter-run agreement already reach SQL, Grafana and the maintainer recap, but never PostHog. So cost, model and latency were readable there while whether the review was any good was not — and the two halves could not be joined.
This emits them as
$ai_metricagainst the review's own trace, so quality lands beside the spend and the model that produced it. With the trace grouping from #10187, a metric on that trace id sits directly alongside the generations it is measuring.The contract comes from the SDK, not from a guess
@posthog/core'scaptureTraceMetric:The value is stringified by the SDK, so this stringifies it too — a hand-built metric and an SDK-built one are indistinguishable downstream. A test pins that, including that a numeric
0and afalsebecome"0"and"false"rather than being dropped: a "did not flag" vote is real signal, not an absent one.Two deliberate refusals
UNCORROBORATED_AGREEMENT(0.5) is a deliberate placeholder for "nothing to agree with", not a measurement — publishing it would drop a fabricated 0.5 into an average alongside real scores. That decision lives in a pure helper beside the scorer (judgmentAgreementMetrics), so both sides of it are testable directly instead of requiring a full dual-reviewer run to reach.Only the confidence-independent half of the agreement score is reported.
agreementandsampleCountare properties of the stances themselves;confidencefolds in a per-finding verbalized confidence, so it belongs to a finding rather than to the review.Closes #10226
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateAlso run and green:
npm run dead-exports:check—POSTHOG_AI_METRIC_EVENTis referenced from the tests, followingPOSTHOG_MONITOR_HEARTBEAT_EVENT's existing pattern.If any required check was skipped, explain why:
A coverage gap that changed the design
The first version guarded the agreement emit with
if (!agreementSignal.uncorroborated)inline in the orchestrator. That branch's true side was unreachable from the existing test harness, which drives a single-reviewer review. Rather than bolt on a dual-reviewer fixture purely to reach it, the decision moved intojudgmentAgreementMetrics— a pure function returning an empty list when there is nothing to report. The orchestrator now just iterates whatever it is given, and both sides are covered by direct unit tests. Better factoring, not a workaround.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.The metric carries a name, a number and the already-processed operational context — no prompt, completion or finding text. Context routes through the shared operational allowlist, so an unlisted key is dropped (a test asserts it), and under the shared central key the repo is HMAC-anonymized and fail-closed exactly as every other event on this path. No reward, trust-score or private-scoring value is emitted — the metrics here are a reviewer's binary stance and an inter-run agreement ratio.
The emission is best-effort alongside the existing audit write, matching that call site's established discipline: the capture is a no-op when PostHog is off and never throws, so it cannot affect a review.
UI Evidence
Not applicable — no visible UI, frontend, docs, or extension change.
Notes
$ai_feedbackand$ai_evaluationare deliberately not included, and the issue says why in full. In short:$ai_feedbackis PostHog's user-feedback event, whose documented path is the Surveys integration — there is no human feedback at these call sites, and dressing a machine vote as one would misrepresent what the event means.$ai_evaluationhas no property contract in the SDK or the docs beyond a one-line taxonomy description, so any shape chosen today would be invented. Worth revisiting once PostHog documents it, or once online evaluations are enabled — which content capture (#10218) unblocks.Four further sources named in the issue (self-consistency, routing shadow, gate precision, eval score records) are left for follow-up; this lands the helper plus the two highest-value call sites.