Expand test analytics to report on test analytics for "ALL Branches"

[rohan-at-sentry]

What product do you want to improve?
Test analytics from Codecov

Is your feature request related to a problem? Please describe.
Customers that experience flakes on CI runs likely don't want to search by individual branches to find aggregate counts of failures across ALL branches (main is also a branch).

Our current table data and stats rollups that are planned (see https://www.figma.com/design/TcppABudxnslgEmaXm0B9O/GH-2028?node-id=1-2&node-type=canvas&t=yLQARHqMTAuV6tZ1-0) are branch specific

Describe the solution you'd like
Showing totals for failures across all branches and average run times / failure rates across all branches would be cool - especially for tests that are known flaky

Write up - https://www.notion.so/sentry/Show-Test-Analytics-across-all-branches-1188b10e4b5d80d6880cc83a4f640251?pvs=4

[rohan-at-sentry]

Some other thoughts

1. Given that the tests dashboard on the main branch lists flake rate and failure rate on main - there isn't really an easy way to answer the question of how many flaky tests are impacting development in aggregate

2. We could change how flake rate or failure rate is calculated on main (since that's the only branch flake rate is shown). This would break consistency of the UI (we show all stats on a per branch basis) - that doesn't seem like a good thing to do

That's sort of why I'm leaning towards a "show all branches" type of roll-up. In that way we can have a view that shows stats for ALL branches in aggregate.

• It preserves our existing model
• We could have it selected by default

[Adal3n3]

We currently have "All branches" for commits, so I'm trying to learn about the scope of this request. @ajay-sentry @joseph-sentry do you know?

[joseph-sentry]

@Adal3n3 I think that's only for coverage at the moment, I don't think we have this for test analytics yet, and it may require some work both on the backend and frontend to get it working for test analytics

[rohan-at-sentry]

Folks asking for this in our feedback repo - codecov/feedback#304 (comment)

[Adal3n3]

Slack thread 9/27: https://sentry.slack.com/archives/C05Q10P3NG5/p1727440512777729
External customer feedback:

Our biggest pain point these days are tests that flake on PRs (pre-merge), and those are the metrics we would want to track at the aggregate level. As a result, we would derive the most value if this dashboard included all non-default branches, if that makes sense. From this view, it looks like we are only able to select one Branch Context at a time.
With those goals in mind, is there a way in Codecov Test Analytics to aggregate all of these results across our branches, or do we need to execute this test+upload cycle on our default branch for the highest fidelity aggregate view?

Abhijeet Prasad's feedback:

I’d like to chime in to say that aggregating on all non-main branches (effectively all pr branches) would be super useful for us because it directly shows impact of tests on dev velocity.
In our case we have multiple “production” branches so we’d want to be able to identify multiple branches (main, release, beta etc.)

Id want a multi-select dropdown where I’d pick a set of branches to explicitly exclude. I don’t want the SDKs release branches (because we maintain multiple majors + alpha/beta versions at a time) to be included in the aggregate calculations. This is because to me repo the definition of a pr branch is not every non-default branch

[rohan-at-sentry]

@katia-sentry - I added a start / end date for this.

The key insight here is

• we're gonna want some of the perf improvments for TA to land (Test Analytics Performance Improvements #2682) before we take this work on
• the UI work is limited to adding a field on a dropdown and sending a query down to backend to "remove all branch filters" which shouldn't take more than a few days.