[Failing Test] Strange ci-benchmark-scheduler-perf-master behavior

[macsko]

Which jobs are failing?

ci-benchmark-scheduler-perf-master

Which tests are failing?

Interestingly, we get benchmark results for all of the test cases, but the job is somehow failing (see testgrid).

Since when has it been failing?

6th September. No changes in scheduler_perf since 26th August.

Testgrid link

https://testgrid.k8s.io/sig-scalability-benchmarks#scheduler-perf

Reason for failure (if possible)

I have no idea what's going on with the test. Tests give the correct results and populate the perf-dash dashboard.

Anything else we need to know?

No response

Relevant SIG(s)

/sig scheduling

[macsko]

cc: @pohly @sanposhiho

[pohly]

/assign

The first failed test case is:

=== FAIL: test/integration/scheduler_perf BenchmarkPerfScheduling/SchedulingCSIPVs/5000Nodes_5000Pods (unknown)

That should have caused the per-test log output to be preserved, which might have had more information, but it's not under the artifacts.

Something isn't quite right with output handling - will check.

[pohly]

SchedulingCSIPVs/5000Nodes_5000Pods has 48 as threshold. In https://storage.googleapis.com/kubernetes-jenkins/logs/ci-benchmark-scheduler-perf-master/1833184953530060800/build-log.txt, it has:

name                                                  SchedulingThroughput/Average
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                 351 ± 0%

name                                                  SchedulingThroughput/Perc50
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                 383 ± 0%

name                                                  SchedulingThroughput/Perc90
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                 509 ± 0%

name                                                  SchedulingThroughput/Perc95
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                 515 ± 0%

name                                                  SchedulingThroughput/Perc99
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                 518 ± 0%

name                                                  runtime_seconds
PerfScheduling/SchedulingBasic/5000Nodes_10000Pods-6                                                44.1 ± 0%

Why is the threshold so low?

... never mind, wrong test.

[pohly]

#125534 changed how go test output is processed. The difference between 0e4cf67..2049360 (working/not working) includes that.

I think we now hit gotestyourself/gotestsum#413 (comment).

[pohly]

I submitted gotestyourself/gotestsum#438. If this doesn't get merged soonish, it's probably better to revert #125534 and re-apply it later with a new gotestsum.

[pohly]

@macsko : how much longer are you willing to wait before we revert?

There have been some discussions around how to fix this in gotestsum, but no conclusion yet.

[macsko]

It's not that urgent until something bad happen with scheduling performance and we don't get an alert.