Would there be any interest in more k8s "native" tooling for integration testing?

[vaikas]

Hey there :)

We in the Sigstore utilize Trillian as our Merkle tree (Thank you!!). As part of our desire to be able to run more extensive and comprehensive e2e testing, we're utilizing kind and utilizing containers for all of our testing. I have put together a sigstore-scaffolding that contains various components that need to be spun up for us to test Sigstore. One of these is the need to create a Trillian (and supporting operations, like creating 2 trees in Trillian). This may be related to what's looked for in #2329 @pavelkalinnikov ?

It's not ideal to have these be in a 'side' repo, and therefore, I wanted to see if there would be any interest in me upstreaming the Trillian related pieces into this repo. It would not be all of these, but just the components related to Trillian.

One nice side benefit also is that you can use 'kind' locally (assuming you have docker) and have a higher degree of confidence that what you're testing with will then also work in ci/cd as well as deployment to k8s, since the constructs are the same.

If there's interest, there's the overview of how we use the k8s constructs (like configmaps / jobs) to communicate the state between components that need information from Trillian (like, treeid created for example).

Anyways, I've yammered too long. Please let me know if there's any interest in this and I'd be happy to work with you and if there are any questions, comments or concerns, I'd be happy to answer them :)

Overview (for Sigstore, but contains information about Trillian, since we need it for both Rekor / CTLog)
https://github.com/vaikas/sigstore-scaffolding/blob/main/README.md

Configurations related to Trillian specific components:
https://github.com/vaikas/sigstore-scaffolding/tree/main/config/trillian

One example run in github actions:
https://github.com/vaikas/sigstore-scaffolding/runs/5097036599?check_suite_focus=true

I think the pieces that would be maybe interesting to upstream would be:

• createtree that can update a configmap in a k8s cluster, plus a job that it runs as.
• github workflow that creates a kind cluster, spins up logserver/logsigner and performs some operations on it
• not sure if there's anything else at this point.

[mhutchinson]

Hello! Yeah this sounds cool and I think bringing these components into this repo would be great. I'm happy to review any PRs you want to send. The other addition to the list would obviously be a README that explains how to use these :-)

In terms of organizing this, we have top level directories for deployment and examples/deployment that are similar. Perhaps for now examples/deployment/kind would be the best place to organize these new files?

[vaikas]

Fantastic! Thanks for getting back to me.
I also saw (and heavily cribbed the existing cmd/createtree) and produced a version that stashes the treeID created into a ConfigMap (so dependant pieces (like CTLog, Rekor) can utilize it in k8s fashion).
https://github.com/vaikas/sigstore-scaffolding/blob/main/cmd/trillian/createtree/main.go

I reckon that might live in:
cmd/createtree-k8s (?)

Another option would be to make it a flag to existing, but seems like that might not be desireable since it will take a dependency on k8s libraries?

I'm totes fine with the examples/deployment/kind but it's not only specific to kind (since any cluster will do). Wonder if examples/deployment/kubernetes/scaffolding might make it more clear that it can be used with other tooling.
Or something like that, I'm terrible with naming, and hence fine with kind in the first place.

Really that piece is only meant to allow Trillian dependent software be able to bootstrap by creating the tree using a k8s job (therefore lending itself nicely to e2e testing on k8s).

Thoughts?

[mhutchinson]

Looks like you have a good grasp on naming; i.e. it's important, but it's hard!

I'm happy with the suggestion for cmd/createtree-k8s. We can roll them into one in the future if we like, but I think on balance I would prefer the duplication of a small amount of code than adding more options and dependencies to this small cmd.

Let's try examples/deployment/kubernetes/scaffolding and see how it looks. I'd originally suggested something other than this directory because a) I've never deployed it using the instructions in there; and b) that directory looked otherwise self-contained and single-purpose. That said, having all the k8s stuff grouped together is going to keep the codebase a lot more navigable. Really this is a signal that I should take some time to play with the trillian on k8s stuff :-)

[mhutchinson]

Any updates here? :-)

[vaikas]

Yes, thank you for the ping and sorry for the tardy reply. I upstreamed my code to sigstore and it now lives here:
https://github.com/sigstore/scaffolding

So, that's also been cleaned up a bit along the way which is nice :)

I'll make time this week to create the PR for it.

[vaikas]

@mhutchinson ok, finally got to it :)