Skip to content

Commit 5e70e39

Browse files
KSemenenkoclaude
andcommitted
Allow each measured benchmark case 15 minutes
The owner raised the per-case time limit from 6 to 15 minutes. It still applies the same way to every database, and loads still always finish. Verified: Debug build of KeyLoad.ComparisonHost with warnings as errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent f3dc385 commit 5e70e39

3 files changed

Lines changed: 6 additions & 7 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ cases, schedule and verification, and the driver interface exist once, and each
306306
database, KeyLoad included, implements only its driver. Change a dataset or
307307
scenario once, in its module. KeyLoad runs every data type. Every database uses
308308
seed 1729, 16 clients, 1,000,000-record datasets and 100,000 planned operations
309-
per case with the same 6-minute limit per measured case (loads always finish), at one
309+
per case with the same 15-minute limit per measured case (loads always finish), at one
310310
node and, where the database supports it, three nodes.
311311
Unsupported capabilities and node counts have no cell; a failed case keeps its
312312
real error; never estimate, zero or invent a number.

‎benchmarks/KeyLoad.ComparisonHost/Measurement/Execution/BenchmarkSettings.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@ internal sealed record BenchmarkSettings(int Records, int Operations, int Client
1313
internal const int StandardClients = 16;
1414
internal const int StandardSeed = 1729;
1515

16-
/// <summary>The mean measured case time across every database of the first per-type run (41203341021): 5.8 minutes.</summary>
17-
internal const int StandardCaseMinutes = 6;
16+
/// <summary>The owner's maximum wall time for one measured case, the same for every database.</summary>
17+
internal const int StandardCaseMinutes = 15;
1818

1919
/// <summary>Spreads <paramref name="index"/> deterministically over <c>[first, first + count)</c>.</summary>
2020
internal int Spread(int index, int first, int count)

‎docs/Features/BenchmarkComparisons.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ decision is [ADR-150](../ADR/ADR-150-per-type-benchmark-jobs.md).
1111
|---|---|
1212
| REQ-BC-030: one job per database, data type and node count | `benchmarks.yml` has one `measure` job per row of the [type table](#data-types); a job deploys one database and runs only the TUnit class of its type. |
1313
| REQ-BC-031: one shared module per data type | The dataset, scenario (cases, schedule, verification) and driver interface of a type exist once; a database implements only the driver. |
14-
| REQ-BC-032: the same standard size everywhere | Seed 1729, 16 concurrent clients, 100,000 planned operations per measured case, at most 6 minutes per measured case, and 1,000,000-record datasets, unless a local run overrides them. Dataset loads are never stopped by the limit. |
14+
| REQ-BC-032: the same standard size everywhere | Seed 1729, 16 concurrent clients, 100,000 planned operations per measured case, at most 15 minutes per measured case, and 1,000,000-record datasets, unless a local run overrides them. Dataset loads are never stopped by the limit. |
1515
| REQ-BC-033: results survive failure | The job writes its result file after startup, after every case and on disposal, and uploads it whether the job passed or failed. |
1616
| REQ-BC-034: no invented numbers | A failed case keeps its real first error; a case that never ran, a missing node count or an unsupported database has no number. |
1717
| REQ-BC-035: the website shows one run | Website publishes the result files of the dispatched run, or of the latest completed run; without result files it publishes nothing and the current Pages stay. |
@@ -136,9 +136,8 @@ latency per call over successful calls; 0 when no call succeeded), `batchSize`
136136
fails when any unit fails or its after-the-fact verification finds a problem, and
137137
the TUnit test fails with it. No value is ever estimated or filled in.
138138

139-
The case time limit is the mean measured case time across every database of the
140-
first per-type run (5.8 minutes, rounded to 6). It applies the same way to every
141-
database: once it has passed, clients take no new operation, operations already
139+
The case time limit is 15 minutes, set by the owner. It applies the same way to
140+
every database: once it has passed, clients take no new operation, operations already
142141
started finish, and `opsPerSecond` and the latencies cover only what ran. The
143142
site shows such a case as "<ran> of <planned>". Loads always finish, because the
144143
following cases read the loaded data. `Benchmarks__CaseMinutes` overrides the

0 commit comments

Comments
 (0)