Youki builds for custom executors

[0xE282B0]

TLDR;
I want to use Youki to run containerd-wasm-shims with CRI-O.

Problem description

Youki supports different runtimes using the executor trait. The current runtimes are WasmTime, WasmEdge, Wasmer. These runtimes are also implemented by the runwasi project, which makes them available as containerd shims, where the Unix implementation is based on libcontainer and also uses the executor trait.

There are other shims based on runwasi, and the deislabs/containerd-wasm-shims repo contains shims for slight, spin, wws, and lunatic.

Issue summary

• Containerd shims only work with containerd
• Youki can be used by other CRIs such as CRI-O, but only supports WasmTime, WasmEdge, and Wasmer.
• To add more runtimes to youki, the executor implementations must be added to the youki crate.

Possible solutions

The easiest solution to run additional runtimes with youki is to copy the executor implementation into the youki crate. But this is not desirable for several reasons, like it would duplicate code and increase maintenance work for the youki crate.

What I'd prefer is to build a specialized version of youki that uses a custom executor. This approach is inspired by how the runwasi shims build their binaries. There the main function is a minimal call to runwasi that configures the shim instance.

Unfortunately, this requires the youki crate to be a library that can be imported into the project where the executor is implemented. To demonstrate how this works, I created a branch based on youki 0.2.0 that converts the youki crate into a lib. In this commit:

• main.rs is moved to lib.rs
• main function is renamed and add a parameter for the executor is added
• bin section in Cargo.toml is created to build youki binary

Example

With that changes the main function of the youki crate is looking like this:

use anyhow::Result;
use libcontainer::workload::default::DefaultExecutor;
use youki::run_youki;

fn main() -> Result<()> {
    run_youki(DefaultExecutor{})
}

To test this approach I created a branch in the containerd-wasm-shims repository. The PR c8fe9ffa shows the changes necessary to build the youki-spin binary which is a build of youki with a spin executor. The main function for it looks like this:

use youki::run_youki;
use containerd_shim_wasm::{sandbox::Stdio, container::executor::Executor};
use engine::SpinEngine;

mod engine;

fn main() -> Result<()> {
    run_youki(Executor::new(SpinEngine{}, Stdio::init_from_std()))
}

Questions

• Would it be ok to propose such a change as a PR?
• Do you have other suggestions to create a OCI compliant runtime from an executor trait?

[utam0k]

@0xE282B0 Thanks for your proposal.
We already have the API that you can create Executors you want. Is it not enough in your case?
e.g.
https://github.com/containers/youki/blob/1be3353835e9c8eeb755852833d816be56c0c9bd/crates/youki/src/commands/create.rs#L17
https://github.com/containers/youki/blob/1be3353835e9c8eeb755852833d816be56c0c9bd/crates/youki/src/workload/executor.rs#L57

[0xE282B0]

Hey @utam0k, yes I use it extensively with libcontainer. But in this case I want to compile an OCI compliant runtime like the youki binary but with a spin executor to use it with CRI-O.
Instead of copying the executor to the youki crate I want to use youki in the spin crate.

It could look like #2429.

[utam0k]

I think the libcontainer crate is sufficient. It is easy to imagine that users will probably ask youki crate(lib) for various parameters. It should not be only Executors that the users want to pass as a parameter.

[0xE282B0]

I know that this change is only for one purpose and I understand that it is not the scope of youki to make OCI runtimes from executors other that the already included.

Yes, the libcontainer crate is sufficient to run implementaions of the executor trait. But to call it from CRI-O/Podman I need to reimplement/copy the cli commands of the youki crate or is there a way to import them?

[YJDoc2]

@Overflow0xFFFF we have liboci-cli crate which would provide the clap impl for oci spec compliant command params for runtime,but you will have to copy over quite a bit of code that is currently in youki binary crate.

[neersighted]

Hey all, I just want to clarify an important point here: what is being described is not a library to build "OCI runtimes" -- that is actually what the current libcontainer is (and I think that's what @utam0k meant). What you are describing is instead what we informally call a "runc-compatible binary."

It's important to keep in mind that the runtime-spec only specifies high-level operations which the runc CLI interface happens to closely resemble. As such (and given that naming of e.g. libcontainer already causes confusion the runc maintainers semi-regularly dispel), I'd suggest it be made clear what is actually being exported to prevent people from being disappointed/confused.

All that being said, the runc interface is not completely implementation-defined, a (outdated) subset exists at https://github.com/opencontainers/runtime-tools/blob/master/docs/command-line-interface.md for example. Still, I think it needs to be clear that "real" implementations will need more than is in the spec (e.g. to support the new concept of features) and that this is not part of the runtime-spec itself/not a "standard" feature of OCI runtimes.

There's certainly value in building binaries which something that knows how to interface with runc can interoperate with, however I think it should made very clear what is being provided and that this is not the runtime CLI (because there is none).

TL;DR: I think liboci-cli is misleadingly named and it should be renamed if at possible, to try to reduce confusion for those hacking in this space.

[utam0k]

@neersighted Thanks for your comment!
I've learned from this issue that we need to sort out our documentation for our users. We already have a good place to describe them.
@neersighted @0xE282B0 May I ask you to help us update our documentation from the user side?
https://containers.github.io/youki/user/crates.html

At least almost all functions used from containerd are written in the oci runtime spec. The days of runc having original functions are ending. This is an important meaning of youki's existence. It is important to maintain the specification properly, and youki does that well. Youki's contribution in this respect is significant. In fact, I am also a specification maintainer.

runc-compatible binary

Maybe you know, but I tell that the oci runtime spec also defines not only CLI interface but also the behavior of the linux containers,
https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md

It's important to keep in mind that the runtime-spec only specifies high-level operations which the runc CLI interface happens to closely resemble.

Sorry but it is hard to rename it because it is used widely.

TL;DR: I think liboci-cli is misleadingly named and it should be renamed if at possible, to try to reduce confusion for those hacking in this space.

[neersighted]

I tell that the oci runtime spec also defines not only CLI interface
I'm still not sure what this mean -- indeed, the OCI runtime spec defines many high-level operations and Youki does a good job of implementing most of them.

However, the exact interface in code used to invoke these operations is only part of runtime-tools, and even then it's a very reduced subset.

Indeed, the runtime spec also encodes elements of the kernel-userspace interface, which has lead to some level of debate as to how "pure" it is and how to interpret certain fields which are influenced by kernel implementation.

Would love to pick your brain on some of the above (the "purity" and Linux-isms) at some point, if you have time.

[utam0k]

Indeed, the runtime spec also encodes elements of the kernel-userspace interface, which has lead to some level of debate as to how "pure" it is and how to interpret certain fields which are influenced by kernel implementation.

I am puzzled this is being written here. Shouldn't this properly be discussed in the specifications? What would be your request to youki? I wasn't sure, sorry ... 😭 Can I ask you to be more specific?

[neersighted]

Sorry, it's not directly related to Youki -- it was meant to be an observation of what is part of runtime-spec, and what is not. My real point is that the command line interface is not part of runtime-spec, and I would find a rename (of liboci-cli) helpful as this is a very common source of confusion.

[utam0k]

What names do you prefer?

[neersighted]

I'd suggest something like librunc-cli or similar, since that is what is really being modeled. I'd say "runc" and not something more generic as it's still the definitive version of the CLI interface, and other programs that are compatible (e.g. Kata v1, crun) are used with the containerd runc shim.

[0xE282B0]

Hi,
Since providing a runc-compliant CLI for arbitrary implementations of the libcontainer Executor trait is outside the scope of the youki binary crate, this issue can be closed.
There seems to be no immediate need for this, but if in the future there is a need to create a runc-compliant CLI for a libcontainer executor, this can be done using the crates provided by the youki project and the youki binary crate as an example.