selfhost(observability): Langfuse LLM tracing — per-review prompt/response quality observability

[JSONbored]

Part of #1029.

Context

The review engine makes AI calls but there is zero visibility into what was sent, what came back, how many tokens were used, what the latency was, or whether the verdict was high-quality. Langfuse is an open-source LLM observability platform — it logs every prompt + response as a "trace" with scores, latency, token cost, and a web UI. Adding it as --profile observability (alongside Prometheus/Grafana) closes the quality feedback loop.

Easy on/off

# Enable:
docker compose --profile observability up -d
# Add to .env:
# LANGFUSE_PUBLIC_KEY=pk-lf-local
# LANGFUSE_SECRET_KEY=sk-lf-local
# LANGFUSE_HOST=http://langfuse:3001

# Disable:
# Remove the three LANGFUSE_* vars — the app sends no traces, zero overhead

No tracing code runs when LANGFUSE_HOST is unset. The SDK is lazy-imported; if the env var is absent the import never happens.

Implementation plan

docker-compose.yml (extend --profile observability)

langfuse:
  image: langfuse/langfuse:latest
  profiles: ["observability"]
  restart: unless-stopped
  ports:
    - "3001:3000"
  environment:
    DATABASE_URL: ${LANGFUSE_DB_URL:-postgresql://langfuse:langfuse@langfuse-db:5432/langfuse}
    NEXTAUTH_SECRET: ${LANGFUSE_SECRET:-changeme-langfuse}
    SALT: ${LANGFUSE_SALT:-changeme-salt}
    LANGFUSE_INIT_ORG_ID: selfhost
    LANGFUSE_INIT_PROJECT_ID: gittensory
    LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_PUBLIC_KEY:-pk-lf-local}
    LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_SECRET_KEY:-sk-lf-local}
  depends_on: [langfuse-db]

langfuse-db:
  image: postgres:16-alpine
  profiles: ["observability"]
  restart: unless-stopped
  environment:
    POSTGRES_USER: langfuse
    POSTGRES_PASSWORD: langfuse
    POSTGRES_DB: langfuse
  volumes:
    - langfuse-db-data:/var/lib/postgresql/data

src/selfhost/langfuse.ts

// Lazy wrapper — zero cost when LANGFUSE_HOST is unset
export async function tracedAiCall<T>(
  name: string,
  fn: () => Promise<T>,
  meta: { model: string; repo: string; pr: number }
): Promise<T> {
  if (!process.env.LANGFUSE_HOST) return fn();
  // import and wrap with Langfuse SDK trace
}

Wrap the createSelfHostAi call sites so every review AI call emits a Langfuse trace with:

• Prompt (sanitized — no raw diff content unless LANGFUSE_INCLUDE_PROMPTS=true)
• Model name + provider
• Token usage + latency
• Gate verdict (approve/block/comment)
• Repo + PR number (for correlation with outcome data)

.env.example

# Langfuse LLM tracing (--profile observability, alongside Prometheus/Grafana)
# LANGFUSE_HOST=http://langfuse:3001
# LANGFUSE_PUBLIC_KEY=pk-lf-local
# LANGFUSE_SECRET_KEY=sk-lf-local
# LANGFUSE_INCLUDE_PROMPTS=false   # set true to log full prompts (review them for PII first)

Acceptance criteria

• docker compose --profile observability up starts Langfuse at port 3001
• http://localhost:3001 shows Langfuse UI with the gittensory project pre-created
• LANGFUSE_HOST set → AI calls emit traces; visible in UI with model/latency/tokens
• LANGFUSE_HOST unset → zero overhead, no import, no network calls
• Both arms of the if (!process.env.LANGFUSE_HOST) branch covered
• LANGFUSE_INCLUDE_PROMPTS=false default prevents raw prompt logging
• npm run test:ci green, Codecov ≥ 97% patch