Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: cibench workflow #179

Merged
merged 7 commits into from
Sep 1, 2024
Merged

Conversation

AaronO
Copy link
Contributor

@AaronO AaronO commented Sep 1, 2024

First pass at adding benchmarks in CI.
Covers matrix of key archs and SIMD extensions.
Outputs delta against main with critcmp (with 5% noise threshold).

Future improvements

  • Run classifier on results to indicate regression/improvement/neutral (aggregate or on specific arch)
  • Post summary (or details if regression) as PR comment (didn't add cause without summary would be noisy)
  • Could save main baselines as assets (and just ref/download them) instead of recomputing them for each PR
    (comes at a cost of statefulness and some adhoc complexity but would reduce CI time cost)

Notes

  • This adds CI (compute) time, but don't think it adds wall time since there are longer subtasks
  • CI is still more noisy than controlled local env, but probably good enough for substantial perf changes and sanity-checking

@AaronO AaronO marked this pull request as ready for review September 1, 2024 05:08
Copy link
Owner

@seanmonstar seanmonstar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@seanmonstar seanmonstar merged commit 38878a0 into seanmonstar:master Sep 1, 2024
41 checks passed
@AaronO AaronO deleted the perf/cibench branch September 1, 2024 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants