Skip to content

feat(stats): AI-feature cache/reuse-efficiency rate, tracked over time, public #4448

Description

@JSONbored

Part of #4445.

Context

Cache-hit/reuse instrumentation exists for only 4 of the ~10 AI-touching capabilities: ai_review (github_app.ai_review_paused_reuse, github_app.ai_review_frozen_reuse, github_app.ai_review_cache_miss), ai_slop (github_app.ai_slop_cache_hit/_miss, plus the new github_app.ai_slop_auto_review_skipped commit-threshold cap), linked_issue_satisfaction (github_app.linked_issue_satisfaction_cache_hit/_miss, src/queue/processors.ts:7934-7947, backed by the linkedIssueSatisfactionCache table, src/db/schema.ts:1432-1451), and miner_detection (github_app.miner_detection_cache_hit/_miss, src/queue/processors.ts:14916-14971).

Six capabilities have zero cache/reuse instrumentation at all: enrichment, grounding, impact-map, repo-culture-profile, reputation, and review-memory (checked enrichment-wire.ts, grounding-wire.ts, impact-map-wire.ts, repo-culture-profile-wire.ts, reputation-wire.ts, review-memory-wire.ts directly — none exist). This means there is currently no way to answer, in aggregate, "what fraction of our AI work was avoided by correctly reusing prior results vs. redundantly redone" — a genuine, honest engineering-efficiency story that's directly downstream of tonight's incident work (the ai_slop repeat-spend bug this epic's context comes from was only caught by hand-querying audit_events for exactly this kind of signal).

This is not a cost/token-rate metric (out of scope per the parent epic) — it's a measure of how well the engine avoids doing the same work twice, which is squarely something we built and can be proud of independent of any specific LLM's pricing.

Requirements

  • Extend cache-hit/miss (or an equivalent "did we do fresh work or reuse a prior result" signal) instrumentation to the six currently-uninstrumented capabilities, matching the existing *_cache_hit/*_cache_miss audit-event naming convention.
  • Build an aggregate "reuse rate" computation: (cache hits + correctly-skipped-as-already-done) / (total invocation attempts), across all ~10 capabilities, per project and fleet-wide.
  • Persist a periodic (daily, matching the other two sub-issues' cadence) rollup rather than computing this live from raw audit_events on every request — the whole reason this needs a rollup table is that hand-querying audit_events live is exactly the failure mode that caused this quarter's incidents.
  • Extend GET /v1/public/stats with the reuse-rate trend (reuse the existing response shape).
  • Surface it publicly with an honest framing: "how often we avoid redoing AI work" — not a cost claim, a competence claim.

Deliverables

  • Cache-hit/miss (or equivalent reuse-signal) instrumentation added to enrichment, grounding, impact-map, repo-culture-profile, reputation, and review-memory
  • Aggregate reuse-rate computation across all ~10 AI-touching capabilities
  • Daily rollup persisted, cron-refreshed
  • GET /v1/public/stats extended with the reuse-rate trend; OpenAPI regenerated
  • Homepage//stats page shows the reuse-rate trend
  • Tests: new instrumentation per capability (both hit and miss branches), aggregate rollup computation, public endpoint shape

Expected outcome

A real, historical, publicly-visible "how efficiently gittensory avoids redundant AI work" number, covering every AI-touching capability instead of the 4 that happen to have this today — and a rollup table that would have made tonight's ai_slop repeat-spend incident visible in dashboards instead of requiring a live production DB query to find.

References

  • src/queue/processors.ts:7934-7947 (linked_issue_satisfaction cache-hit/miss, one of the 4 existing examples to mirror)
  • src/queue/processors.ts:14916-14971 (miner_detection cache-hit/miss, another existing example)
  • src/db/schema.ts:1432-1451 (linkedIssueSatisfactionCache table, an existing cache-table pattern to mirror where a new one is needed)
  • src/services/enrichment-wire.ts, grounding-wire.ts, src/services/impact-map-wire.ts, repo-culture-profile-wire.ts, reputation-wire.ts, review-memory-wire.ts (the six capabilities needing new instrumentation)
  • src/review/public-stats.ts:179-346, src/api/routes.ts:949-957 (existing public stats endpoint to extend)
  • This quarter's ai_slop repeat-spend fix (merged, runAiSlopForAdvisory's new commitThresholdReached cap) — the concrete incident this rollup would have surfaced without a manual DB query

Effort

L

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions