Proposal for a runtime package.py entry and OCI support

[Minkiu]

With the advent of containerization, I've been wondering if we could add support for this. I do want to still be using rez and I've been thinking how it could fit within this.

The idea would allow users to declare a new key in their packages runtime (I am not set on this name nor any of the others, suggestions welcome) which in turn relies in new rezconfig entries:

# oci = open container initiative https://opencontainers.org/
oci_daemon = 'podman' # or 'docker' maybe even apptainer?
oci_repository = https://hub.docker.com/ # or option for users to have their one one first, I guess depending on the above setting, you get one or other... (or maybe it's at distro level)
oci_whitelist_images = ['aswf/ci-base:2018'] # Kind of a railguard to prevent users from arbitrarily pull images

And in the DCC/tool package.py we could then have something like:
runtime="aswf/ci-base:2019"

So that then when we do rez-env foo_runtime, rez would then:

1. resolve environment
2. pull image
3. check for or create pet container (rootless) called awsf_ci_base_2019 # We want to reuse the containers

Now either:

4. Option A: podman exec -it awsf_ci_base_2019 bash -c "rez-env foo_runtime"
   Notes: Some check needs to be performed to know if we are in a pet container to avoid an infinite loop..

Or:

4. Option B: podman exec -it --env-file /path/to/env/file awsf_ci_base_2019 bash
   Notes: Inherit the whole environment either map all of them with --env MY_ENV=MY_VALUE or pass a --env-file

Considerations:

• At the minute none (correct me if I am wrong) of the AWSF are pulling the nvida-opengl image which would allow for GUI apps within this environments (end goal for me really) - https://github.com/NVIDIA/nvidia-docker
• runtimes conflict unless they are (all) the exact same one. Lets not get into overlayfs territory just yet..
• no-runtime == good ol' bash (or whatever shell you have configured)
• There needs to be a way to somehow specify network mappings, maybe we should allow runtime=docker-compose.yml or runtime=Dockerfile driven by settings in rezconfig

So for a host, all you need (I'm gravely simplifying it) is a OS that has python and OCI containers support. Big inspiration is comming from toolbx if anyone is curious.

Let me know if this is something we could benefit from having in rez or if it's way out of scope...

[maxnbk]

What happens when more than one package has a runtime defined? Must a requirement of this be that a solve possess a single runtime/entrypoint?

I've been thinking about containers and rez being more tightly coupled, but I think that the approach is best-used outside the package-domain. i.e, something like rez-run that allows you to "attempt" to run a given entrypoint inside a given container, but the solve is separate from the container. That said, there is a need to ensure that builds could build inside of a given container (IMO), so that adds some complexity as well.

[Minkiu]

I'd say it would be similar to variants, if two packages are trying to run in different runtimes it would error, so we catch this before we go and create more containers.

[nerdvegas]

This seems like it should be a separate project no? Something that associates rez packages with images, and is responsible for booting you into the right container+rez_runtime when requested. It feels like something outside the scope of this project.

…

On Thu, Feb 10, 2022 at 7:34 AM maxnbk ***@***.***> wrote: What happens when more than one package has a runtime defined? Must a requirement of this be that a solve possess a single runtime/entrypoint? — Reply to this email directly, view it on GitHub <#1200 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSW5HONS2JWPOSV7F7TU2LFUVANCNFSM5N6KJ53A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Minkiu]

Well part of the idea was to not to have to come up with yet another layer, but get your point.

I mean I could easily now do:

toolbox create --image aswf/ci-base:2019 my_container
toolbox enter my_container
rez-env ...

Maybe it could be more of a hinting thing? so that the other package can benefit of this through the API? 🤔

[maxnbk]

I think there could be something to be said for rez potential involvement. Perhaps solving for the best-possible-runtime is a valid concern? at the very least, a tight coupling could be beneficial. I've long wanted a way to express to rez, "perform this build in this container", and I could immediately see that becoming "perform this build in this container, but each variant needs to build in a different container". Expressing that as a function of def build_command(): seems inadvisable.

[JeanChristopheMorinPerso]

I've long wanted a way to express to rez, "perform this build in this container"

Wound't that be covered by the build process plugins? Right now there is only local and remote, but I guess a docker plugin could be added?

[nerdvegas]

Actually yes, when considering build runtimes this is applicable, so it could make sense. But, consider the different options we have for supporting this: 1. We could still avoid the tight coupling with rez completely, and implement this build workflow in a separate project. Rez has a rich enough API to support this, although we'd probably still find changes to make to minimise the amount of external code you'd have to write to get this working. 2. We aim for the more ambitious idea of extensible runtimes. Right now when you construct a runtime, you're put into a subshell where a bunch of shell script has been sourced (mostly made up of setting env-vars, but not exclusively). and these generally refer to paths into package payloads stored in their repos, typically on shared disk somewhere. However we could support different implementations, such as overlayfs-based (to avoid issues such as long PYTHONPATH); symlink-based (where you dynamically generate symlinks to simulate the idea of an overlaid fs); and so you could add a container-based runtime too, where the image to use is chosen through some mechanism related to the package request being resolved. I like this approach because it solves much more than building stuff in a container, however the scope of work will not be trivial. A

…

On Thu, Feb 10, 2022 at 8:01 AM maxnbk ***@***.***> wrote: I think there could be something to be said for rez potential involvement. Perhaps solving for the best-possibly-runtime is a valid concern? at the very least, a tight coupling could be beneficial. I've long wanted a way to express to rez, "perform this build in this container", and I could immediately see that becoming "perform this build in this container, but each variant needs to build in a different container". Expressing that as a function of def build_command(): seems inadvisable. — Reply to this email directly, view it on GitHub <#1200 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSQKC2AYCVS67FP4PP3U2LI2JANCNFSM5N6KJ53A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[Minkiu]

Option 2 feels a bit like Nix packaging system? Is rez's ambition to become something like that?

I like this approach because it solves much more than building stuff in a container

That's on purpose (building stuff in a container), so everything runs in the same "OS" even tho then the rez-env is different. My idea with this is to be able to run any arbitrary OS that supports python and containers (so we don't hold back the whole desktop experience for the sake of "stability"), not saying you didn't get that from my initial post, just reiterating my point :D

[nerdvegas]

Imo we should think of rez as like the LLVM of packaging. It's supposed to be a modular and extensible way to package and consume software, I think that's the goal. So for example, for a given package you should be able to build it onto the filesystem, or into an image; then consume it in various different ways depending on the runtime engine you've chosen (which doesn't exist yet - I'm spitballing here).

The point I'm making in (2) is less about any specific implementation and more about the fact it should be extensible in general. You should be able to manage packages in a container-based way if you want to, or just on the filesystem instead, and that shouldn't make any difference to the packages' source/definition - they should be agnostic to how they end up deployed.

This is already true for:

• OS - the same package defn can deploy onto linux/mac/win
• Package definition storage (aka package repo) - you could implement a postgres-based repo for eg, or a REST service-based one;
• Shell - package definitions express their "commands" as python specifically so that multiple shells are supported when configuring the runtime (there are 6 supported currently)
• Build system (this one is a bit of a grey area since clearly if you're using say cmake, you need cmake source. But the supported build systems are plugin based and you can add more)

Imo extensibility of the runtime itself is a natural progression of the existing design. I just wish I'd thought of it sooner :)

[maxnbk]

I imagine that the spk crowd might have something to say about the overlay-fs approach. :)

[Minkiu]

Pardon the ignorance (and my gogle-fu is not returning much) but what would be spk ?

[instinct-vfx]

Imageworks' Package manager

[christian-korneck]

JFWIW there’s another container related use case that hasn’t been mentioned yet: Using oci container images to store package payload. This means at build time to build and push a “from scratch” container image (per package version/variant) and at resolve time to pull and mount the image on the host (if not already present). The mount path can be used like a regular path (local,nfs,etc) to the package payload. Main benefits are local payload caching (alternative to rez’s existing payload caching) and making rez packages (at least the payload part) instantly usable for cloud rendering and cloud workstations by being able to use a container registry for it.

There are countless possible alternatives to this (i.e. just copying zip files from/to s3), but the broader cloud ecosystem seems to have decided for oci images as the canonical format for application bundles and there’s a broad, healthy and still evolving ecosystem with many tools and services around the apparently stable oci format (i.e. large cloud providers offer managed container registries with geo replication).

(A side effect of such a setup is that these container images for individual package payloads can be relatively easily used to build larger container images for other use cases, ex: centos:7 + package1:1.0 + package2:2.0.2. Container image layers make this efficient).

(With all that said, this is only about the package payloads, the package “metadata” side needs to be handled separately).

[nerdvegas]

Actually that's true, we could absolutely do this as a new build process. A

…

On Fri, Feb 11, 2022 at 9:54 AM Jean-Christophe Morin < ***@***.***> wrote: I've long wanted a way to express to rez, "perform this build in this container" Wound't that be covered by the build process plugins? Right now there is only local and remote, but I guess a docker plugin could be added? — Reply to this email directly, view it on GitHub <#1200 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSW5UVLMXVQ42U7NQMLU2Q6ZLANCNFSM5N6KJ53A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[JeanChristopheMorinPerso]

JFWIW there’s another container related use case that hasn’t been mentioned yet: Using oci container images to store package payload.

Storing build results/payloads into something else than a filesystem is still something that has to be done. I would see a new package repository plugin for zip archives, OCI images, etc.

Building stuff in a container can be done via a build process plugin.

And I think it's maybe what @nerdvegas was referring to (or something similar) when he talks about modularity. We should not focus on one technology, but develop something that will work for multiple technologies.

As for storing runtime details in the package definition files, I don't think it's a good idea personally. Or to be more precise, docker/OCI specific things shouldn't be defined as top level attributes. Maybe a runtime attribute that is a dict and can contain anything you want, but I wouldn't go with oci_daemon, etc.