Skip to content

feat(enrichment): flaky-test history annotator analyzer #2033

Description

@JSONbored

New REES github-heavy analyzer (the #1499 epic 'flaky-test history annotator' idea) that, for test files touched by the PR, surfaces whether those tests appear in recent CI check-run failure history for the base branch — so a reviewer knows a change lands on a historically-flaky test. Uses the GitHub check-runs/commits API with strict fanout caps; returns partial/degraded on budget exhaustion; fail-safe without a token.

Deliverables

  • Add FlakyTestFinding ({ file, recentFailures: number, window: string }) + flakyTest? key to review-enrichment/src/types.ts (counts only, no logs)
  • Create review-enrichment/src/analyzers/flaky-test.ts: for changed test files, query bounded recent check-run history and count failures referencing that file/suite (injected fetch); strict maxFilesProbed + subcall caps; emit partial status when capped
  • Register a descriptor (category 'history', cost 'github-heavy', requires ['files','github-token']) in review-enrichment/src/analyzers/registry.ts with inline render()
  • Add review-enrichment/test/flaky-test.test.ts covering a flaky test flagged, a clean test not flagged, the probe cap → partial status, and the token-absent skip

References

  • review-enrichment/src/analyzers/churn-hotspot.ts (github-heavy per-file bounded history probe template with partial/degraded)
  • review-enrichment/src/analyzers/history.ts (bounded fanout + partial-status pattern)
  • gh issue 1499 lists 'Flaky-test history annotator' as a later idea

Part of #1499.


size: S · gittensor:feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions