New test tooling #3418

apostasie · 2024-09-07T04:17:13Z

We are not in a good situation wrt our testing. The build would be constantly red if it wasn't for the fact we are retrying every test multiple times. Some of it may be bugs, but most of it is definitely broken tests.

Despite lengthy efforts to reduce flakiness one by one, it seems unlikely to me that we can make significant progress without better test tooling to use.

This here proposes a new testing approach that should hopefully address our endemic issues and promote better testing in the future, by providing by default better test isolation, simpler, more debuggable helpers, and overall a more expressive testing approaching.

Commit 1 is the tooling itself.

Commit 2 is a handful of tests being rewritten with the new tooling.

Commit 3 is documentation.

Background information:

the commit message has the full details - read it for more details
this fixes testutil helpers are hard to use and make too many assumptions on command outcomes #3130

Obviously, this should keep evolving a lot, and some stuff might have to change as we rewrite more tests and discover more use-cases.

I would suggest reading the testtool.md doc first before engaging with code.

Right now, what matters most IMHO is that:

we agree on the problem statement - hopefully we do :-)
we are overall comfortable with the high-level goals and design approach (let's call it "table driven test++")
we find the proposed DX acceptable enough

Improvements can always follow.

apostasie · 2024-09-18T05:38:29Z

@AkihiroSuda this is ready for a first review.

CI is pending.

pkg/testutil/test/test.go

AkihiroSuda · 2024-09-18T05:49:07Z

pkg/testutil/test/test.go

+	WithConfig(key ConfigKey, value ConfigValue) Data
+	ReadConfig(key ConfigKey) ConfigValue
+
+	// Private methods


private methods do not need to be defined in the interface?

I do need getLabels and getConfig here:

nerdctl/pkg/testutil/test/data.go

Lines 131 to 137 in f1434f4

func configureData(t *testing.T, seedData Data, parent Data) Data {

if seedData == nil {

seedData = &data{}

}

dat := &data{

config: seedData.getConfig(),

labels: seedData.getLabels(),

Will remove adopt, which is not necessary.

Done (removed adopt).

Lmk if there is a better way to achieve this without defining the other two on the interface.

AkihiroSuda · 2024-09-18T05:50:18Z

pkg/testutil/nerdtest/test.go

+	"github.com/containerd/nerdctl/v2/pkg/inspecttypes/dockercompat"
+	"github.com/containerd/nerdctl/v2/pkg/inspecttypes/native"
+	"github.com/containerd/nerdctl/v2/pkg/testutil"
+	"github.com/containerd/nerdctl/v2/pkg/testutil/test"


Why separate pkg?

These are nerdctl specific (NerdctlToml, etcetera). Motivation was that there is already a lot inside the test package, and the nerdtest package will likely grow a lot more as it subsumes base.

I can merge them back together though if preferable. No strong opinion here.

cmd/nerdctl/main_test_test.go

docs/dev/testtool.md

AkihiroSuda

Thanks

apostasie · 2024-09-18T23:42:26Z

Thanks

@AkihiroSuda

I forgot to convert some tests that are commented out.
Let me take care of that ASAP before we merge.

Our testing tools have served us well, and there are plenty of good things with them that we absolutely need to keep. We have grown a large test suite though, and the sheer size of it is putting pressure on it and it is starting to show cracks. Besides bugs per-se (double execution) - that we hopefully fixed - our design is reaching its limits. We do have rather impactful issues with regard to test isolation and test concurrency, leading to situations where it is hard or outright impossible to figure out which test is causing a cascading failure, or why the adding of a new test file working individually breaks everything. Furthermore, as we are not prescriptive on certain things, we do see a lot of negative patterns emerging from test authors: - defer being used instead of t.Cleanup - not calling cleanup routines before running tests - outright forgetting to cleanup resources - Cwd for binary calls is by default the current working directory of the test process - this is causing a variety of issues, as it could very well be read-only (lima), and should by default be a temp directory - manipulating the environment directly - which has side-effects for other tests - tests becoming big blurbs of mixed together setup, cleanup, and actual test routines - making them hard to read and figuring out what is actually being tested - subtests repetitiveness w. shadowing testutil.T leading to confusing code - ... or not dereferencing the current test in a loop - in-test homegrown abstractions being inherently repetitive, with the same boilerplate over and over again - structuring tests and subtests being left as an exercise to the developer - leading to a wide variety of approaches and complex boilerplate - hard to debug: a lot of the assert methods do not provide any feedback whatsoever on what was the command that actually failed, or what was the output, or environment of it - icmd.Expected showing its limits, and making assumptions that there is only one error you can test, or that you can only test the output if exitCode == 0 - running commands with other binaries than the current target (eg: not calling base.Cmd) being left as an exercise to the developer, leading to all of the issues above (Chdir, Env manipulation, no debugging output, etc) - no-parallel being the default - unless specified otherwise - which should be the other way around - very rarely testing stderr in case of error - partly because we do not use typed errors, but also because it is cumbersome / not obvious / limited This new tooling offers a set of abstractions that should address all of these and encourage more expressive, better structured, better isolated, more debuggable tests. Signed-off-by: apostasie <[email protected]>

Signed-off-by: apostasie <[email protected]>

apostasie · 2024-09-19T23:02:40Z

@AkihiroSuda last push finish converting and re-enabled the mistakenly commented out tests.
Apologies for missing it.

Also note a small change in the test tooling, enabling Parallel for test.Group.

apostasie · 2024-09-20T00:36:53Z

CI failures are network flukes, which have been endemic with github recently.
Concerning for sure, but unrelated.

apostasie force-pushed the dev-3130-tesutil-nextgen branch 4 times, most recently from 8cb9f25 to 814418a Compare September 7, 2024 22:37

apostasie force-pushed the dev-3130-tesutil-nextgen branch 4 times, most recently from 37bc5e9 to 193a67d Compare September 18, 2024 05:30

apostasie changed the title ~~[WIP] Test tooling refresh~~ New test tooling Sep 18, 2024

apostasie marked this pull request as ready for review September 18, 2024 05:34

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

pkg/testutil/test/test.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

cmd/nerdctl/main_test_test.go Show resolved Hide resolved

AkihiroSuda reviewed Sep 18, 2024

View reviewed changes

docs/dev/testtool.md Outdated Show resolved Hide resolved

apostasie force-pushed the dev-3130-tesutil-nextgen branch 2 times, most recently from 6b60abc to 28263b1 Compare September 18, 2024 06:32

AkihiroSuda approved these changes Sep 18, 2024

View reviewed changes

AkihiroSuda added the area/ci e.g., CI failure label Sep 18, 2024

AkihiroSuda marked this pull request as draft September 18, 2024 23:44

apostasie added 3 commits September 19, 2024 15:58

Move root and volume tests to new test tooling

5048eb3

Signed-off-by: apostasie <[email protected]>

Testtool initial documentation

4a27229

Signed-off-by: apostasie <[email protected]>

apostasie force-pushed the dev-3130-tesutil-nextgen branch from 28263b1 to 4a27229 Compare September 19, 2024 23:01

apostasie marked this pull request as ready for review September 19, 2024 23:02

apostasie requested a review from AkihiroSuda September 19, 2024 23:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New test tooling #3418

New test tooling #3418

apostasie commented Sep 7, 2024 •

edited

Loading

apostasie commented Sep 18, 2024

AkihiroSuda Sep 18, 2024

apostasie Sep 18, 2024

apostasie Sep 18, 2024

AkihiroSuda Sep 18, 2024

apostasie Sep 18, 2024

AkihiroSuda left a comment

apostasie commented Sep 18, 2024

apostasie commented Sep 19, 2024 •

edited

Loading

apostasie commented Sep 20, 2024

	func configureData(t *testing.T, seedData Data, parent Data) Data {
	if seedData == nil {
	seedData = &data{}
	}
	dat := &data{
	config: seedData.getConfig(),
	labels: seedData.getLabels(),

New test tooling #3418

Are you sure you want to change the base?

New test tooling #3418

Conversation

apostasie commented Sep 7, 2024 • edited Loading

apostasie commented Sep 18, 2024

AkihiroSuda Sep 18, 2024

Choose a reason for hiding this comment

apostasie Sep 18, 2024

Choose a reason for hiding this comment

apostasie Sep 18, 2024

Choose a reason for hiding this comment

AkihiroSuda Sep 18, 2024

Choose a reason for hiding this comment

apostasie Sep 18, 2024

Choose a reason for hiding this comment

AkihiroSuda left a comment

Choose a reason for hiding this comment

apostasie commented Sep 18, 2024

apostasie commented Sep 19, 2024 • edited Loading

apostasie commented Sep 20, 2024

apostasie commented Sep 7, 2024 •

edited

Loading

apostasie commented Sep 19, 2024 •

edited

Loading