Commit 693aed8
authored
fix(observability): group AI generations by their real trace, and attribute spend per repo (#10187)
* feat(analytics): complete the PostHog migration — retire Umami, fix the MCP event contract
Umami is fully decommissioned and PostHog is the only analytics sink.
Web analytics (#8293, #8299). posthog-js replaces the hand-rolled Umami
beacon in routes/__root.tsx, behind the same first-party /stats proxy the
Umami collect endpoint used. lib/analytics.ts is the single seam; the SDK is
loaded via a dynamic import so an unconfigured build pays zero bytes, matching
browser-sentry.ts. Env is read at call time, not module scope, so the token
gate is stubbable. Umami parity was audited against the removed beacon's actual
payload: posthog-js supplies url/referrer/screen/language automatically,
page_title is added explicitly (it has no $title equivalent), geo still comes
from the proxy's x-forwarded-for, and respect_dnt replaces the beacon's own
doNotTrack check. persistence is "localStorage", not "memory" — memory resets
identity every reload and inflates unique visitors, the regression
JSONbored/metagraphed#8210 had to correct. cookieless_mode was rejected on its
documented behavior: it strips the IP before GeoIP runs, which would drop the
country data #8299 requires.
The proxy keeps its allowlist, one level looser (path prefixes, not exact
paths) so a posthog-js upgrade cannot silently break capture. Its original
justification was Umami-specific — that host served its admin API from the
collect origin — which does not carry over, but bounding a public
unauthenticated route still does. Session-replay snapshots get their own 2 MiB
ceiling so enabling #8295 cannot silently 413.
One behavior deliberately changes: the beacon fired once per page load and
never on SPA navigation, so pageview counts run legitimately higher after
cutover.
MCP telemetry (#10175). $mcp_tool_call was emitting LoopOver's own snake_case
property names under PostHog's reserved event name, so their built-in MCP
dashboards ingested every event and rendered every breakdown empty. It now
carries the canonical $mcp_* properties, with LoopOver's surface/transport/
category alongside them (PostHog's custom-server docs sanction extra
properties, and those three have no canonical equivalent). usage_event is
untouched — it stays the minimal, payload-free LoopOver event in its own
vocabulary, and the allowlist meta-test now checks the two vocabularies
separately.
Adds the two missing canonical events: $mcp_initialize, sourced from the
handshake's own clientInfo rather than the x-loopover-mcp-* headers that only
our published client sets, and $mcp_tools_list, reporting what this server
registered. Both carry $session_id from Mcp-Session-Id, which is what makes a
funnel across handshake, discovery, and tool calls possible at all. The closed
LoopOver error-code set is projected onto PostHog's closed $mcp_error_type set
so their error breakdown populates without losing the precise code.
Reshaping the wire format is free right now and stops being free immediately:
the project holds zero $mcp_tool_call events, because POSTHOG_API_KEY was never
set on the Worker.
Prettier (#10176). A root prettier run found no config, fell back to the
80-column default, and rewrapped backend source that nothing in CI formats —
repeatedly, inside unrelated PRs. Adds a default-deny root .prettierignore that
re-admits only the three governed workspaces, restates the generated-file and
content/docs exemptions that a root run cannot inherit, and pins ui-kit to its
current effective width so it is stated rather than inherited from a vendor
default. Verified by running prettier --write . from the root and confirming a
no-op. No existing code is reformatted.
Also formats two ui-kit test files that were already failing their own
format:check on main, which would otherwise have turned this PR red.
Closes #8293
Closes #8299
Closes #10175
Closes #10176
* fix(observability): group AI generations by their real trace, and attribute spend per repo
Two faults made PostHog's AI observability structurally unable to answer the
questions that matter most for a product where AI is the product.
Every generation was its own trace. capturePostHogAiGeneration set
$ai_trace_id to a fresh randomUUID() per call, producing 13,428 AI events
across 13,428 distinct trace ids on the live project. A review fans out across
RAG embeddings, both dual-review legs, N retries per model, and any
self-consistency runs — and each landed as an unrelated single-event trace, so
what a review actually did, and where its time and money went, was not
recoverable.
The fix was already in the file. operationalProperties has called
currentOtelTraceIds() since #8296 and attached the real ids as plain
trace_id/span_id properties; the line below then minted a UUID and PostHog
grouped by that instead. withReviewPipelineSpan already wraps a whole review,
so the ambient trace id is exactly the grouping key wanted — every provider
attempt inside one review now nests under one PostHog trace. $ai_span_id and a
readable $ai_span_name come from the same source.
randomUUID() stays as the fallback rather than being removed: AI_EMBED,
AI_VISION and AI_ADVISORY run outside any review span, so an orphan trace is a
legitimate outcome there — but such an event must not claim a span id it does
not have, so $ai_span_id is omitted rather than faked.
AI spend was unattributable. Every AI event carried distinct_id
"loopover-selfhost" and no $groups at all, so cost per repository was
answerable only by SQL against ai_usage_events, never in PostHog. A repo group
is now stamped on every event. It deliberately reads the already-processed repo
value off operationalProperties rather than the raw context: under the shared
LOOPOVER_CENTRAL_POSTHOG_KEY that value is HMAC-anonymized, and is dropped
entirely when the anon secret has not been injected, so the group inherits that
fail-closed behavior for free and a raw private repo name cannot reach the
group index by this path.
Both branches of both changes are covered: ambient span vs none, and repo
present vs dropped.
Content capture ($ai_input/$ai_output_choices) is deliberately untouched — the
metadata-only policy is documented in self-hosting-operations.mdx and changing
it is a privacy decision, not a wiring one.
Closes #101851 parent d4441f7 commit 693aed8
2 files changed
Lines changed: 77 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
417 | 417 | | |
418 | 418 | | |
419 | 419 | | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
420 | 436 | | |
421 | | - | |
422 | | - | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
423 | 443 | | |
424 | 444 | | |
425 | 445 | | |
| |||
440 | 460 | | |
441 | 461 | | |
442 | 462 | | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
443 | 470 | | |
444 | 471 | | |
445 | 472 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
601 | 601 | | |
602 | 602 | | |
603 | 603 | | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
604 | 652 | | |
605 | 653 | | |
606 | 654 | | |
| |||
0 commit comments