This repository provides a GitHub Action for integrating Nyrkiö Change Detection with your Continuous Benchmarking Results.
This action takes a file that contains benchmark output. The benchmark results are parsed into a common JSON format, which is then sent to nyrkio.com for analysis. All the main benchmarking frameworks are supported.
Let's start with a minimal workflow setup. For explanation, here let's say we have a Go project. But basic setup is the same when you use other languages. For language-specific setup, please read the later section.
Note: You need to create an account at nyrkio.com to get the NYRKIO_JWT_TOKEN that is used below. To receive alerts from Nyrkiö as pull request comments or GitHub issues, the recommended way to create your account is to install Nyrkiö as a GitHub app.
name: Minimal setup
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
benchmark:
name: Performance regression check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v4
with:
go-version: "stable"
# Run benchmark with `go test -bench` and stores the output to a file
- name: Run benchmark
run: go test -bench 'BenchmarkFib' | tee output.txt
- name: Analyze benchmark results with Nyrkiö
uses: nyrkio/change-detection@v2
with:
tool: 'go'
output-file-path: output.txt
# Pick up your token at https://nyrkio.com/docs/getting-started
nyrkio-token: ${{ secrets.NYRKIO_JWT_TOKEN }}The step which runs nyrkio/change-detection does followings:
- Extract benchmark result from the output in
output.txt - Transform the benchmark results into a common JSON format
- Send the new result(s) to Nyrkiö API for analysis
- If a change is detected, either immediately or after a few days, you can receive alerts by slack, email, and github issues.
A live workflow example is here. And the output of the workflow can be seen here and finally the graphs on Nyrkiö here.
If you've managed to automate and run some benchmarks routinely as part of your GitHub Action pipelines, you may have noticed it is not easy to spot performance regressions (nor improvements) in all those benchmark results your GitHub Action workflows now produce! Nyrkiö was developed to help with this last bit. We will analyze your history of benchmark results, and alert you if we find any regressions. (Or improvements, for that matter!)
Automating the analysis of continuous benchmarking results is a surprisingly difficult task. Many benchmarks have a range of variation - random noise - anywhere from 5 % to 20 %. Nyrkiö uses a state of the art change detection algorithm to find statistically significant, persistent changes, in noisy benchmarking results. It adapts automatically to the noise level in your benchmarks, thus avoiding false positive alerts. By analyzing the entire history of results, it can detect even very small regressions, in particular, changes smaller than your range of random noise!
You can read more about how Nyrkiö works here.
This GitHub Action supports the commonly used benchmarking frameworks of all major programming
languages. In addition we support some generic options such as the time utility, a custom
JSON format, and Nyrkiö's own JSON format in pass-thru mode:
cargo benchfor Rust projectsgo test -benchfor Go projects- benchmark.js for JavaScript/TypeScript projects
- pytest-benchmark for Python projects with pytest
- Google Benchmark Framework for C++ projects
- Catch2 for C++ projects
- BenchmarkTools.jl for Julia packages
- Benchmark.Net for .Net projects
- benchmarkluau for Luau projects
- JMH for Java projects
- time Unix utility to measure execution time of any program
- Custom benchmarks where either 'biggerIsBetter' or 'smallerIsBetter'
- Nyrkios own JSON format, which is simply passed directly to Nyrkiö API.
Multiple languages in the same repository are supported for polyglot projects.
We wish to thank the open source community for this incredible feat of integration!
Nyrkiö Change Detection GitHub Action is based on 6 years of work over at the
"GitHub Action Benchmark". Nyrkiö will be upstreaming patches like a
good open source citizen, whenever we add functionality for our own customers. (Starting with
the time command support.)
Example projects for each language are in examples/ directory. Live example workflow definitions are in .github/workflows/ directory. Live workflows are:
| Language | Workflow | Example Project |
|---|---|---|
| Rust |  |
examples/rust |
| Go |  |
examples/go |
| JavaScript |  |
examples/benchmarkjs |
| Python |  |
examples/pytest |
| C++ |  |
examples/cpp |
| C++ (Catch2) |  |
examples/catch2 |
| Julia |  |
examples/julia |
| .Net |  |
examples/benchmarkdotnet |
| Java |  |
examples/java |
| Luau | Coming soon | Coming soon |
You can also simply send your benchmark results to Nyrkiö in Nyrkiö's own JSON format. An example is included in the same format as above.
Nyrkiö JSON is documented in the main Nyrkiö documentation.
This JSON format created and supported by the github-action-benchmark team.
Additionally, even though there is no explicit example for them, you can use
customBiggerIsBetter and customSmallerIsBetter to use this
action and create your own graphs from your own benchmark data. The name in
these tools define which direction "is better" for your benchmarks.
For more details, please consult the LEGACY_README
Nyrkiö can comment on your PR. when benchmark results changed too much compared to the main branch. For this to work you need to install Nyrkiö as a github app into your org. This allows Nyrkiö to send comments on your pull requests and file github issues. Please click on https://github.com/apps/nyrkio/installations/new and proceed to install Nyrkiö if you didn't already.
- name: Analyze benchmark results with Nyrkiö
uses: nyrkio/change-detection@v2
with:
tool: 'go'
output-file-path: output.txt
# Comment on the pull request if there were changes in the benchmark results, but do not hard fail
comment-on-alert: true
fail-on-alert: false
nyrkio-token: ${{ secrets.NYRKIO_JWT_TOKEN }}Now, instead of making workflow fail, the step leaves a comment on the PR when it detects performance regression like this.
Please read README.md files at each example directory. Usually, take stdout from a benchmark tool
and store it to file. Then specify the file path to output-file-path input.
cargo benchfor Rust projectsgo testfor Go projects- Benchmark.js for JavaScript/TypeScript projects
- pytest-benchmark for Python projects with pytest
- Google Benchmark Framework for C++ projects
- catch2 for C++ projects
- BenchmarkTools.jl for Julia projects
- Benchmark.Net for .Net projects
- benchmarkluau for Luau projects - Examples for this are still a work in progress.
These examples are run in workflows of this repository as described in the 'Examples' section above.
Input definitions are written in action.yml.
- Type: String
- Default:
"Benchmark"
Name of the benchmark. This value must be identical across all benchmarks in your repository.
- Type: String
- Default: N/A
Tool for running benchmark. The value must be one of "cargo", "go", "benchmarkjs", "pytest",
"googlecpp", "catch2", "julia", "jmh", "benchmarkdotnet","benchmarkluau", "customBiggerIsBetter", "customSmallerIsBetter".
- Type: String
- Default: N/A
Path to a file which contains the output from benchmark tool. The path can be relative to repository root.
Note: When using tool: nyrkioJson format, this parameter should instead be a directory with JSON files.
- Type: String
- Default: N/A
GitHub API access token. Can be used to get meta-data about your pull request or commit.
- Type: Boolean
- Default:
false
If it is set to true, this action will leave a pull request comment both if there was a regression
and if there wasn't.
- Type: Boolean
- Default:
false
If it is set to true, this action will leave a comment on your pull request
when an alert happens like this.
Note: Currently with Nyrkiö this is synonymous with comment-always
- Type: Boolean
- Default:
false
If it is set to true, the workflow will fail when an alert happens. Note that because Nyrkiö also
alerts on positive changes (such as fixing a regression), then setting this to true together
with using strict branch protection in GitHub may not be that practical.
- Type: String
- Default: nullS
A JWT token from your nyrkio.com settings page: https://nyrkio.com/docs/getting-started
- Type: Boolean
- Default:
false
Publish these benchmark results at nyrkio.com/public. (Recommended for open source projects.)
- Type: String
- Default: ""
The name of a github org where you have installed Nyrkiö. See https://nyrkio.com/user/settings#nyrkio-settings-userinfo
- Type: Number
- Default: 0.001
The p-value used to determine if a change in benchmark results is statistically significant. Example: 0.01 (If specified, will be applied to all your nyrkio.com test results.)
- Type: String (percentage)
- Default: 5%
Threshold percentage below which changes will be ignored. Example: 2% (If specified, will be applied to all your nyrkio.com test results.)
- Type: Boolean
- Default:
false
Don't fail the workflow, ever. Not even if the action itself fails due to bugs or network or whatever.
- Type: Boolean
- Default:
true
Use Nyrkiö (web service) to store your benchmark results and to find statistically significant change points.
- Type: String (URL)
- Default: https://nyrkio.com/api/v0/
The root URL for a Nyrkiö API
The remaining parameters aren't used by Nyrkiö. We've preserved the functionality of the original threshold based alerting system, but its use is not recommended. The related parameters are documented in LEGACY_README.md.
No action output is set by this action for the parent GitHub workflow.
This action conforms semantic versioning 2.0.
For example, nyrkio/change-detection@v2 means the latest version of 2.x.y. And
nyrkio/[email protected] always uses v2.0.2 even if a newer version is published.
HEAD would use the most recent commit of the development branch. This is for experimental and
development use only!
To notice new version releases, please watch 'release only' at this repository. Every release will appear on your GitHub notifications page.