fix(observability): group AI generations by their real trace, and attribute spend per repo

Two faults made PostHog's AI observability structurally unable to answer the
questions that matter most for a product where AI is the product.

Every generation was its own trace. capturePostHogAiGeneration set
$ai_trace_id to a fresh randomUUID() per call, producing 13,428 AI events
across 13,428 distinct trace ids on the live project. A review fans out across
RAG embeddings, both dual-review legs, N retries per model, and any
self-consistency runs — and each landed as an unrelated single-event trace, so
what a review actually did, and where its time and money went, was not
recoverable.

The fix was already in the file. operationalProperties has called
currentOtelTraceIds() since #8296 and attached the real ids as plain
trace_id/span_id properties; the line below then minted a UUID and PostHog
grouped by that instead. withReviewPipelineSpan already wraps a whole review,
so the ambient trace id is exactly the grouping key wanted — every provider
attempt inside one review now nests under one PostHog trace. $ai_span_id and a
readable $ai_span_name come from the same source.

randomUUID() stays as the fallback rather than being removed: AI_EMBED,
AI_VISION and AI_ADVISORY run outside any review span, so an orphan trace is a
legitimate outcome there — but such an event must not claim a span id it does
not have, so $ai_span_id is omitted rather than faked.

AI spend was unattributable. Every AI event carried distinct_id
"loopover-selfhost" and no $groups at all, so cost per repository was
answerable only by SQL against ai_usage_events, never in PostHog. A repo group
is now stamped on every event. It deliberately reads the already-processed repo
value off operationalProperties rather than the raw context: under the shared
LOOPOVER_CENTRAL_POSTHOG_KEY that value is HMAC-anonymized, and is dropped
entirely when the anon secret has not been injected, so the group inherits that
fail-closed behavior for free and a raw private repo name cannot reach the
group index by this path.

Both branches of both changes are covered: ambient span vs none, and repo
present vs dropped.

Content capture ($ai_input/$ai_output_choices) is deliberately untouched — the
metadata-only policy is documented in self-hosting-operations.mdx and changing
it is a privacy decision, not a wiring one.

Closes #10185

Author: JSONbored

src/selfhost/posthog.ts

@@ -417,9 +417,29 @@ export type PostHogAiGenerationEvent = {
  *  parallel concern to error tracking, not a substitute gate for it. */
 export function capturePostHogAiGeneration(event: PostHogAiGenerationEvent): void {
   if (!active || !client) return;
+  const operational = operationalProperties(event.context);
+  // #10185: the OTel trace this call already runs inside IS the trace PostHog should group by.
+  // operationalProperties has computed it (as plain `trace_id`/`span_id`) since #8296, but this
+  // function then minted a fresh randomUUID() for $ai_trace_id and threw it away -- so every
+  // generation became its own single-event trace and the review pipeline that produced them was
+  // never visible as one thing (13,428 events across 13,428 traces on the live project).
+  //
+  // withReviewPipelineSpan (./review-tracing.ts) wraps a whole review, so every provider attempt
+  // inside it -- both dual-review legs, every retry, every self-consistency run, the RAG embeddings
+  // -- shares that span's trace id and now nests under one PostHog trace.
+  //
+  // randomUUID() stays the fallback, NOT an error: AI_EMBED/AI_VISION/AI_ADVISORY and any call made
+  // outside a review span legitimately have no ambient trace, and an orphan trace is still a valid
+  // (if less useful) event. `$ai_span_id` is only set when there IS a real span to name.
+  const traceId = typeof operational.trace_id === "string" ? operational.trace_id : undefined;
+  const spanId = typeof operational.span_id === "string" ? operational.span_id : undefined;
   const properties: Record<string, unknown> = {
-    ...operationalProperties(event.context),
-    $ai_trace_id: randomUUID(),
+    ...operational,
+    $ai_trace_id: traceId ?? randomUUID(),
+    ...(spanId === undefined ? {} : { $ai_span_id: spanId }),
+    // Names the node in the trace tree. Provider-and-kind rather than the tool/feature name, because
+    // that is what this function actually knows -- the feature lives in ai_usage_events, not here.
+    $ai_span_name: `ai.${event.requestKind}/${nonBlank(event.provider) ?? "unknown"}`,
     $ai_model: nonBlank(event.model) ?? "unknown",
     $ai_provider: nonBlank(event.provider) ?? "unknown",
     // PostHog's own $ai_generation schema reports latency in SECONDS, not ms.
@@ -440,6 +460,13 @@ export function capturePostHogAiGeneration(event: PostHogAiGenerationEvent): voi
     distinctId: POSTHOG_DISTINCT_ID,
     event: event.requestKind === "embedding" ? "$ai_embedding" : "$ai_generation",
     properties,
+    // #10185: a real group, so "which repo costs the most in AI spend" is answerable in PostHog
+    // rather than only in the ai_usage_events SQL table. Deliberately reads the ALREADY-PROCESSED
+    // `repo` off operationalProperties rather than event.context: under the shared central key that
+    // value is HMAC-anonymized, and when the anon secret has not been injected the key is dropped
+    // entirely. Reusing it means the group inherits that fail-closed behavior for free -- a raw
+    // private repo name can never reach the group index by this path.
+    ...(typeof operational.repo === "string" ? { groups: { repo: operational.repo } } : {}),
   });
 }
 

test/unit/selfhost-posthog.test.ts

@@ -601,6 +601,54 @@ describe("withPostHogMonitor", () => {
 describe("capturePostHogAiGeneration (#8296)", () => {
   const BASE = { provider: "ollama", model: "llama3.1", requestKind: "review" as const, latencyMs: 1500, isError: false };
 
+  // #10185: trace linking + repo grouping. These are what turn a pile of orphan generations into a
+  // reviewable pipeline and an answerable spend question, so both branches of each are pinned.
+  it("groups the generation under the AMBIENT OTel trace rather than minting a throwaway one", async () => {
+    otelMocks.currentOtelTraceIds.mockReturnValue({ trace_id: "review-trace-1", span_id: "provider-span-1" });
+    await initPostHog({ POSTHOG_API_KEY: "phc_test_key" } as unknown as NodeJS.ProcessEnv);
+    capturePostHogAiGeneration(BASE);
+    const call = mocks.capture.mock.calls[0]?.[0];
+    // The whole point: every provider attempt inside one withReviewPipelineSpan shares this id, so
+    // both dual-review legs, the retries, and the RAG embeddings nest under ONE PostHog trace.
+    expect(call.properties.$ai_trace_id).toBe("review-trace-1");
+    expect(call.properties.$ai_span_id).toBe("provider-span-1");
+    expect(call.properties.$ai_span_name).toBe("ai.review/ollama");
+  });
+
+  it("falls back to a minted trace id, and omits $ai_span_id, when there is no ambient span", async () => {
+    // AI_EMBED / AI_VISION / AI_ADVISORY run outside any review span. An orphan trace is a valid
+    // event, not an error -- but it must not claim a span id it does not have.
+    otelMocks.currentOtelTraceIds.mockReturnValue(undefined);
+    await initPostHog({ POSTHOG_API_KEY: "phc_test_key" } as unknown as NodeJS.ProcessEnv);
+    capturePostHogAiGeneration(BASE);
+    const call = mocks.capture.mock.calls[0]?.[0];
+    expect(call.properties.$ai_trace_id).toEqual(expect.any(String));
+    expect(call.properties.$ai_trace_id).not.toBe("");
+    expect("$ai_span_id" in call.properties).toBe(false);
+  });
+
+  it("names an embedding span by its own request kind", async () => {
+    otelMocks.currentOtelTraceIds.mockReturnValue({ trace_id: "t", span_id: "s" });
+    await initPostHog({ POSTHOG_API_KEY: "phc_test_key" } as unknown as NodeJS.ProcessEnv);
+    capturePostHogAiGeneration({ ...BASE, requestKind: "embedding", provider: "" });
+    expect(mocks.capture.mock.calls[0]?.[0].properties.$ai_span_name).toBe("ai.embedding/unknown");
+  });
+
+  it("stamps a repo group so AI spend is attributable per repository", async () => {
+    await initPostHog({ POSTHOG_API_KEY: "phc_test_key" } as unknown as NodeJS.ProcessEnv);
+    capturePostHogAiGeneration({ ...BASE, context: { repo: "owner/repo", pullNumber: 7 } });
+    const call = mocks.capture.mock.calls[0]?.[0];
+    expect(call.groups).toEqual({ repo: "owner/repo" });
+  });
+
+  it("sends no group at all when no repo survives the operational allowlist", async () => {
+    // Notably the fail-closed central-key path: when the repo is dropped rather than anonymized,
+    // the group must be dropped with it rather than defaulting to some placeholder bucket.
+    await initPostHog({ POSTHOG_API_KEY: "phc_test_key" } as unknown as NodeJS.ProcessEnv);
+    capturePostHogAiGeneration(BASE);
+    expect("groups" in mocks.capture.mock.calls[0]?.[0]).toBe(false);
+  });
+
   it("is a no-op when PostHog is unconfigured", () => {
     capturePostHogAiGeneration(BASE);
     expect(mocks.capture).not.toHaveBeenCalled();