Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

πŸ“Š Adds automatic benchmarking #241

Merged
merged 6 commits into from
Oct 20, 2021
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion .github/workflows/dotnet-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,43 @@ jobs:
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build --configuration Release --no-restore
run: dotnet build -c Release --no-restore
- name: Test
run: dotnet test --no-restore --verbosity normal

benchmark:

name: Run benchmarks
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Setup .NET Core 5.0
uses: actions/setup-dotnet@v1
with:
dotnet-version: 5.0.x
- name: Install dependencies
run: dotnet restore
- name: Build
run: dotnet build -c Release --no-restore
- name: Run benchmarks
run: dotnet run --project Bearded.Utilities.Benchmarks -c Release --no-build -- -f '*' --join --exporters json
- name: Copy benchmark data to deterministic location
run: cp BenchmarkDotNet.Artifacts/results/*-report-full-compressed.json ./new-benchmark-data.json
- name: Download previous benchmark data
uses: actions/cache@v2
with:
path: ./benchmarks
key: ${{ runner.os }}-benchmark
- name: Process benchmark result
uses: Happypig375/[email protected]
with:
tool: 'benchmarkdotnet'
output-file-path: ./new-benchmark-data.json
external-data-json-path: ./benchmarks/benchmark-data.json
fail-on-alert: true
comment-on-alert: true
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Persist benchmark data
if: github.event_name != 'pull_request'
run: cp ./new-benchmark-data.json ./benchmarks/benchmark-data.json
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -187,3 +187,6 @@ FakesAssemblies/

# Rider files
.idea/

# Benchmark outputs
BenchmarkDotNet.Artifacts/
18 changes: 18 additions & 0 deletions Bearded.Utilities.Benchmarks/Bearded.Utilities.Benchmarks.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net5.0</TargetFramework>
<LangVersion>8</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Bearded.Utilities\Bearded.Utilities.csproj" />
</ItemGroup>

</Project>
67 changes: 67 additions & 0 deletions Bearded.Utilities.Benchmarks/Linq/Extensions.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using Bearded.Utilities.Linq;
using BenchmarkDotNet.Attributes;

namespace Bearded.Utilities.Benchmarks.Linq
{
[SuppressMessage("ReSharper", "ClassCanBeSealed.Global")]
public sealed class Extensions
{
private const int seed = 1337;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could potentially lead to inconsistent results if the implementation of Random changes. Did you try just not using a seed to average over a variety of different seeds? (Assuming seeds of Random objects created quickly after each other are random enough - otherwise we could consider generating them with the static random class...)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe using a fixed seed is desirable. You are right that this could lead to inconsistent results if the implementation of Random changes. However, the implementation of Random changing must be because we update a dependency or runtime. In that case, the results being inconsistent in itself becomes an important signal. If we see a significant performance reduction because we update or .NET version, then that is worth finding.

By using a variety of different seeds, we surely average out things correctly, but all of a sudden we have to start tuning our benchmarks to make them (1) as fast as possible while at the same time make them (2) run often enough to balance out the statistical noise from the random.

Using a fixed seed is what is used across the board in the examples in the BenchmarkDotNet documentation. I believe my argument above justifies following that example.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The library examples themselves using static seed surprises me, but perhaps there is indeed a reason... but given that it reruns the test many times, it would still make more sense, at least in principle, to use random seeds to find out NOW if the performance in inconsistent (doesn't the library give us variance measurement?), instead of waiting for a runtime update to surprise us. :D

private const int count = 10000;

public abstract class ListBenchmark
{
protected List<int> List { get; }
protected Random Random { get; }

protected ListBenchmark()
{
List = new List<int>(count);
Random = new Random(seed);
for (var i = 0; i < count; i++)
{
List.Add(Random.Next());
}
}
}

public class RandomElement : ListBenchmark
{
[Benchmark]
paulcscharf marked this conversation as resolved.
Show resolved Hide resolved
public int GetRandomElement()
{
return List.RandomElement();
}
}

public class RandomSubset : ListBenchmark
{
[Params(1, 100, count / 2, count - 100, count)]
public int SubsetSize { get; set; }

[Benchmark]
public List<int> GetRandomSubset()
{
return List.RandomSubset(SubsetSize);
}
}

public class Shuffle : ListBenchmark
{
[Benchmark]
public void ShuffleInPlace()
{
List.Shuffle(Random);
}

[Benchmark]
public IList<int> Shuffled()
{
return List.Shuffled(Random);
}
}
}
}
13 changes: 13 additions & 0 deletions Bearded.Utilities.Benchmarks/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
ο»Ώusing BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;

namespace Bearded.Utilities.Benchmarks
{
static class Program
{
static void Main(string[] args)
{
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, new DebugInProcessConfig());
}
}
}
6 changes: 6 additions & 0 deletions Bearded.Utilities.sln
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bearded.Utilities.Testing",
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bearded.Utilities.Testing.Tests", "Bearded.Utilities.Testing.Tests\Bearded.Utilities.Testing.Tests.csproj", "{D31CF841-95DD-4B6A-887C-930796E7FEDC}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bearded.Utilities.Benchmarks", "Bearded.Utilities.Benchmarks\Bearded.Utilities.Benchmarks.csproj", "{6078D548-38CD-49B4-B1F7-228DB6325E06}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand All @@ -33,6 +35,10 @@ Global
{D31CF841-95DD-4B6A-887C-930796E7FEDC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{D31CF841-95DD-4B6A-887C-930796E7FEDC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{D31CF841-95DD-4B6A-887C-930796E7FEDC}.Release|Any CPU.Build.0 = Release|Any CPU
{6078D548-38CD-49B4-B1F7-228DB6325E06}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{6078D548-38CD-49B4-B1F7-228DB6325E06}.Debug|Any CPU.Build.0 = Debug|Any CPU
{6078D548-38CD-49B4-B1F7-228DB6325E06}.Release|Any CPU.ActiveCfg = Release|Any CPU
{6078D548-38CD-49B4-B1F7-228DB6325E06}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down