Ephemeral packages are here

[nerdvegas]

https://github.com/nerdvegas/rez/wiki/Ephemeral-Packages

Keen to here of real world applications of these, so please post to this discussion if and when anything comes up.

As the release notes hint, ephemerals are also a precursor to 'package features', which will be a way to depend on a package feature other than its version. For example, you may require a debug build of a specific package, as well as its version within a specified range.

[bpabel]

Awesome!

I will definitely use this for package features (like .python-2). So the python-2 package and all the applications with embedded python2 interpreters can include .python-2 in their requires, and then any python packages that need a specific version of python can requires the .python-2 ephemeral package as well.

Are you planning additional changes for package features? Or do you envision it working using requires to specify that a package either provides or needs a feature.

[nerdvegas]

Bear in mind that if you request .python-2, that can still choose variants that don't have that requirement at all (same as a standard request). So if you wanted to use it as a kind of 'package feature', you'd have to ensure that all variants have some .python requirement, and that's probably unrealistic.

This is where "package features" come in. They will be a special case of ephemerals, and variants will have to have that requirement in order to be selected. This is going to require an update to the solver, but not a major one I think.

So for example, you might have this:

# in package.py
name = 'foo'
version = '1.0.0'
variants = [
  [.feature.debug-0],
  [.feature.debug-1]
]

Here we have defined a global "debug" feature. Now if you go:

]$ rez-env foo .feature.debug-1

..then you are guaranteed to get a foo variant with this feature.

The devil will be in the details with this however. For example, package-specific features will be common (what if I want a debug build of just foo, and not other packages). What does that look like? How do I facilitate asking for debug builds of everything, but a non-debug build of foo? Or debug builds of everything, except those things that don't have a 'debug' feature at all?

I think we should just start with the raw features (ie, ephemerals, package features, as described), and go from there. Ie, once these are available and in use, I think we'll start to get a better idea of how to tackle these bigger issues. In the meantime I'll keep thinking about it.

ps - one more thing. Syntax-wise, the above is a little clunky. I think we'll probably end up supporting a new syntax for this stuff, however under the hood it would still translate to what's shown above. So for example, it may end up that a request like foo-1+/debug-1 would be equivalent to [foo-1+, .feature.foo.debug-1]. But again this can come later.

[bpabel]

For the package specific optional features, python uses a bracketed notation foo[debug,test], though, in the python case it has nothing to do with version resolution, it's only considered once a package version has been resolved.

[nerdvegas]

ah yes this is "extras" right? It might be good to lend the same notation.

However, we'd have to be careful about what this means. Someone might take foo[debug] to mean, "I want a debug build of foo". But that isn't what this is saying - it's saying "I want foo with a 'debug' feature", but that feature might be 1 or 0. The important thing here is that you're saying "I do NOT want foo that does NOT have a debug feature". In this case, what's making it confusing is our choice of feature. "debug" is a bad choice - "build_type" would be better, where you'd expect requests like .build_type-debug for eg. In that case it becomes more obvious - foo[build_type] means (more intuitively) "I want a foo that has defined its build type". foo[build_type-debug] means "I want a debug build of foo".

[davidlatwe]

Is there any reason not allow to access ephemerals object in package's late binding functions ?

[nerdvegas]

Certainly not, I missed that!

…

On Mon, 1 Mar. 2021, 20:12 David Lai, ***@***.***> wrote: Is there any reason not allow to access ephemerals object in package's late binding functions ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1001 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSRDJOBSVEU6NUW2HD3TBNLBRANCNFSM4VMZF3HQ> .

[davidlatwe]

Thanks @nerdvegas !

Was looking at the code after I asked the question, and I found although it's good to expose that in late bindings, but attributes like requires may not get any ephemerals since it need to be resolved, correct ?

If so and is expected outcome, it seems not hard to implement ?

[davidlatwe]

In contrast to ResolvedContext._get_pre_resolve_bindings, maybe add ResolvedContext_get_post_resolve_bindings to opt-in both resolve and ephemerals objects into late bindings only after resolved ?

[nerdvegas]

Ah right here's the thing - you can't expose any post-resolve stuff to late bindings. It's a chicken-and-egg situation - late bindings have to be evaluated pre-resolve, because they can affect the resolve itself. So actually, it doesn't make sense to expose `ephemerals` to late-bound funcs, I was incorrect. A

…

On Tue, Mar 2, 2021 at 12:44 AM David Lai ***@***.***> wrote: In contrast to ResolvedContext._get_pre_resolve_bindings, maybe add ResolvedContext_get_post_resolve_bindings to opt-in both resolve and ephemerals objects into late bindings only after resolved ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1001 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSWI26AM3PZ32OH7LP3TBOK4XANCNFSM4VMZF3HQ> .

[ColinKennedy]

I am late to the party but I would like to mention that ephemerals are easily my favorite Rez feature and sets Rez apart from other package managers.

Ephemeral packages are an extra tool in our toolkit that effortlessly provide two capabilities that you'd otherwise have to make extra packages for.

• API backwards compatibility
• Dynamic packages

API backwards compatibility

During a show, there a show-critical Houdini HDA that needed a major reworking that would cause a new major version in the package. That Rez package, unfortunately, was used and updated by many devs, daily, and would've been incredibly disruptive to maintain an old or new major version at the same time. There were talks of splitting it off into a newly named Rez package but then not only would we need to refactor + move existing code into a third "core" package that could be shared but also the new HDA needed to be the same name as the previous HDA and having a separate package increased the chances of breaking Houdini scene files.

Luckily for us, ephemerals released not too long ago and we were able to completely solve the problem without a new major version nor a separate Rez package

Enter, Ephemerals

name = "houdini_hdas"

variants = [[".houdini_hdas.feature.new_hda_api-1", "extra_requires-2.1"], [".houdini_hdas.feature.new_hda_api-0"]]

build_command = "python {root}/rezbuild.py"

And then in the rezbuild.py (this is heavily oversimplified but roughly was like this) ...

import os
import shutil

source = os.environ["REZ_BUILD_SOURCE_PATH"]
destination = os.environ["REZ_BUILD_INSTALL_PATH"]
shutil.copytree(os.path.join(source, "common_hdas"), os.path.join(destination, "hdas"))

if os.getenv("REZ_EPH_HOUDINI_HDAS_FEATURE_NEW_HDA_API", "0") == "1":
    shutil.copytree(os.path.join(source, "new_hda"), os.path.join(destination, "hdas"))
else:
    shutil.copytree(os.path.join(source, "old_hda"), os.path.join(destination, "hdas"))

The new_hda and old_hda contained a Houdini HDA with the exact same name but one was the newer API and the other was the old. For old jobs, we included "houdini_hdas.feature.new_hda_api-0" in the Rez requests list. Newer jobs would default to the new HDA, instead. The HDAs continued to be the same name, we continued releasing new minor / patch Rez package versions, and the two systems coexisted perfectly. Many months later after the old job was over, we deleted the old_hda folder, removed the ephemeral logic and variants, and just made a new minor, officially deprecating the old HDA without disrupting production.

Regular environment variables couldn't really solve this easily because we still needed separate variants with different requirements for each configuration. And that configuration also had to accessible during rez-build. If a user wanted to try to use environment variables to solve this problem, they'd need to remember to set those variables correctly each time they tried to rez-build / rez-release. Ephemerals just work as-is, because they function like regular packages.

Dynamic Packages

I'll start by describing a package I've written in the past and then end with a real package example that you can check out.

Ephemerals As A One-Package-Fits-All Solution

I wrote a validation tool in Python, using Rez. This validation tool has no dependencies at minimum and the core is basically just a graph execution framework, of sorts. Early on in the project, we wanted a PySide GUI so I made two packages, validation_core and validation_gui.

Other departments got wind of this cool validation thing and asked for it to be added to Maya. Not wanting to pollute existing packages with maya-related dependencies, I made another package, validation_maya_plugin, which depended on validation_core and validation_gui.

Later, Houdini support was added, so now there were 4 packages

• validation_core
• validation_houdini_plugin
• validation_maya_plugin
• validation_gui

And then we started adding USD validation, which would then hook into existing plugins, so, not wanting to pullute the DCC-plugins / core with USD dependencies, another Rez package was born

• validation_core
• validation_houdini_plugin
• validation_maya_plugin
• validation_gui
• validation_usd

This went on and on until there were over 7 interrelated packages. And to some extent, this split actually worked alright.

Later on, I needed to make a tweak to validation_core, a package that all other Rez packages depended upon. The rez-test for validation_core passed and I made a rez-release but, without realizing it, I'd broken validation_houdini_plugin. validation_maya_plugin and the others were fine but validation_houdini_plugin failed and if I had ran the rez-test for validation_houdini_plugin before releasing, I would've caught it. It wasn't supposed to be an API compatibility break (and technically wasn't) but the bug occurred anyway and wasn't caught.

We had all of these interconnected packages that needed to be tested prior to each release. It was done manually, it was a pain, and reproducing problems became harder as versions shifted between packages.

Enter, Ephemerals

Those packages were later deleted and combined into a single Rez package. And instead, the package looked like this

@late()
def requires():
    output = [  # base requirements
        "python-2.7+<3.8",
        "schema-0.7+<1",
        "six-1.16+<2",
    ]

    if not in_context():
        return output

    if ".validation.features.usd" in request and intersects(request[".validation.features.usd"], "1"):
        output += ["USD-19.11+<20"]
		
    if ".validation.features.maya_plugin" in request and intersects(request[".validation.features.maya_plugin"], "1"):
        output += ["maya-2018+<2023", "maya_bonus_tools", "other_stuff-1"]

    if ".validation.features.houdini_plugin" in request and intersects(request[".validation.features.houdini_plugin"], "1"):
        output += ["houdini-18.5+<20", "sidefx_labs", "more_stuff-1"]

    return output
	
	
tests = {
    "unittest_core": "python -m unittest discover --start-directory tests_core",
    "unittest_houdini": {
        "command": "hython -m unittest discover --start-directory tests_houdini",
            "requires": [".validation.features.houdini_plugin-1"],
	},	
    "unittest_houdini_usd": {
        "command": "mayapy -m unittest discover --start-directory tests_houdini",
            "requires": [".validation.features.houdini_plugin-1", ".validation.features.usd-1"],
        },
    "unittest_maya": {
        "command": "mayapy -m unittest discover --start-directory tests_maya",
            "requires": [".validation.features.maya_plugin-1"],
        },
    "unittest_maya_usd": {
        "command": "mayapy -m unittest discover --start-directory tests_maya",
            "requires": [".validation.features.maya_plugin-1", ".validation.features.usd-1"],
	},
    "unittest_usd": {
        "command": "python -m unittest discover --start-directory tests_usd",
            "requires": [".validation.features.usd-1"],
    },
    # ... etc etc ...
}

This had several immediate effects

• Reduced our number of packages and releases
• Plugins are now guaranteed to work together (usd and houdini will always work together, for example, because they're under the same version)
• The code was all in one place so rez-build took a bit longer (about 2-3 seconds instead of < 1) however that was mitigated by other systems that were already set in place, such as symlinked local builds.
• most importantly, we got one major upside - If you make changes to the core, you can see whether that change is compatible in every other houdini / maya / USD plugin, etc all in one rez-test call. It made CI a breeze

Conclusion

These are two concrete examples, there are others, Like for example, if you want to distribute a package with different configurations, ephemerals + variants is a great combo. The fact that ephemerals can extend / grow packages as opt-in / opt-out flags is really powerful. It makes packages more resilient and gives you more options when in the past you used to split your code in various places, make extra packages etc or, in other cases, have to make a new major version and deal with the consequences of that.

[nerdvegas]

Just a note - all your ephemeral requests there really start with '.', right? Otherwise they won't be recognised as ephs, but as real packages. Thx A

…

On Thu, Jun 30, 2022 at 4:19 PM Colin Kennedy ***@***.***> wrote: I am late to the party but I would like to mention that ephemerals are easily my favorite Rez feature and sets Rez apart from other package managers. Ephemeral packages are an extra tool in our toolkit that effortlessly provide two capabilities that you'd otherwise have to make extra packages for. - API backwards compatibility - Dynamic packages API backwards compatibility During a show, there a show-critical Houdini HDA that needed a major reworking that would cause a new major version in the package. That Rez package, unfortunately, was used and updated by many devs, daily, and would've been incredibly disruptive to maintain an old or new major version at the same time. There were talks of splitting it off into a newly named Rez package but then not only would we need to refactor + move existing code into a third "core" package that could be shared but also the new HDA needed to be the same name as the previous HDA and having a separate package increased the chances of breaking Houdini scene files. Luckily for us, ephemerals released not too long ago and we were able to completely solve the problem without a new major version nor a separate Rez package Enter, Ephemerals name = "houdini_hdas" variants = [["houdini_hdas.feature.new_hda_api-1", "extra_requires-2.1"], ["houdini_hdas.feature.new_hda_api-0"]] build_command = "python {root}/rezbuild.py" And then in the rezbuild.py (this is heavily oversimplified but roughly was like this) ... import osimport shutil source = os.environ["REZ_BUILD_SOURCE_PATH"]destination = os.environ["REZ_BUILD_INSTALL_PATH"]shutil.copytree(os.path.join(source, "common_hdas"), os.path.join(destination, "hdas")) if os.getenv("REZ_EPH_HOUDINI_HDAS_FEATURE_NEW_HDA_API", "0") == "1": shutil.copytree(os.path.join(source, "new_hda"), os.path.join(destination, "hdas"))else: shutil.copytree(os.path.join(source, "old_hda"), os.path.join(destination, "hdas")) The new_hda and old_hda contained a Houdini HDA with the exact same name but one was the newer API and the other was the old. For old jobs, we included "houdini_hdas.feature.new_hda_api-0" in the Rez requests list. Newer jobs would default to the new HDA, instead. The HDAs continued to be the same name, we continued releasing new minor / patch Rez package versions, and the two systems coexisted perfectly. Many months later after the old job was over, we deleted the old_hda folder, removed the ephemeral logic and variants, and just made a new minor, officially deprecating the old HDA without disrupting production. Regular environment variables couldn't really solve this easily because we still needed separate variants with different requirements for each. If a user wanted to try to use environment variables to solve this problem, they'd need to remember to set those variables correctly each time they tried to rez-build / rez-release. Ephemerals just work as-is, because they function like regular packages. Dynamic Packages I'll start by describing a package I've written in the past and then end with a real package example that you can check out. Ephemerals As A One-Package-Fits-All Solution I wrote a validation tool in Python, using Rez. This validation tool has no dependencies at minimum and the core is basically just a graph execution framework, of sorts. Early on in the project, we wanted a PySide GUI so I made two packages, validation_core and validation_gui. Other departments got wind of this cool validation thing and asked for it to be added to Maya. Not wanting to pollute existing packages with maya-related dependencies, I made another package, validation_maya_plugin, which depended on validation_core and validation_gui. Later, Houdini support was added, so now there were 4 packages - validation_core - validation_houdini_plugin - validation_maya_plugin - validation_gui And then we started adding USD validation, which would then hook into existing plugins, so, not wanting to pullute the DCC-plugins / core with USD dependencies, another Rez package was born - validation_core - validation_houdini_plugin - validation_maya_plugin - validation_gui - validation_usd This went on and on until there were over 7 interrelated packages. And to some extent, this split actually worked alright. Later on, I needed to make a tweak to validation_core, a package that all other Rez packages depended upon. The rez-test for validation_core passed and I made a rez-release but, without realizing it, I'd broken validation_houdini_plugin. validation_maya_plugin and the others were fine but validation_houdini_plugin failed and if I had ran the rez-test for validation_houdini_plugin before releasing, I would've caught it. It wasn't supposed to be an API compatibility break (and technically wasn't) but the bug occurred anyway and wasn't caught. We had all of these interconnected packages that needed to be tested prior to each release. It was done manually, it was a pain, and reproducing problems became harder as versions shifted between packages. Enter, Ephemerals Those packages were later deleted and combined into a single Rez package. And instead, the package looked like this @late() def requires(): output = [ # base requirements "python-2.7+<3.8", "schema-0.7+<1", "six-1.16+<2", ] if not in_context(): return output if "validation.features.usd" in request and intersects(request["validation.features.usd"], "1"): output += ["USD-19.11+<20"] if "validation.features.maya_plugin" in request and intersects(request["validation.features.maya_plugin"], "1"): output += ["maya-2018+<2023", "maya_bonus_tools", "other_stuff-1"] if "validation.features.houdini_plugin" in request and intersects(request["validation.features.houdini_plugin"], "1"): output += ["houdini-18.5+<20", "sidefx_labs", "more_stuff-1"] return output tests = { "unittest_core": "python -m unittest discover --start-directory tests_core", "unittest_houdini": { "command": "hython -m unittest discover --start-directory tests_houdini", "requires": ["validation.features.houdini_plugin-1"], }, "unittest_houdini_usd": { "command": "mayapy -m unittest discover --start-directory tests_houdini", "requires": ["validation.features.houdini_plugin-1", "validation.features.usd-1"], }, "unittest_maya": { "command": "mayapy -m unittest discover --start-directory tests_maya", "requires": ["validation.features.maya_plugin-1"], }, "unittest_maya_usd": { "command": "mayapy -m unittest discover --start-directory tests_maya", "requires": ["validation.features.maya_plugin-1", "validation.features.usd-1"], }, "unittest_usd": { "command": "python -m unittest discover --start-directory tests_usd", "requires": ["validation.features.usd-1"], }, # ... etc etc ... } This had several immediate effects - Reduced our number of packages and releases - Plugins are now guaranteed to work together (usd and houdini will always work together, for example, because they're under the same version) - The code was all in one place so rez-build took a bit longer (about 2-3 seconds instead of < 1) however that was mitigated by other systems that were already set in place, such as symlinked local builds. - most importantly, we got one major upside - If you make changes to the core, you can see whether that change is compatible in every other houdini / maya / USD plugin, etc all in one rez-test call. It made CI a breeze Conclusion These are two concrete examples, there are others, Like for example, if you want to distribute a package with different configurations, ephemerals + variants is a great combo. The fact that ephemerals can extend / grow packages as opt-in / opt-out flags is really powerful. It makes packages more resilient and gives you more options when in the past you used to have to make a new major version and deal with the consequences of that. — Reply to this email directly, view it on GitHub <#1001 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSTSKFSIRSH6IZEDH7LVRU355ANCNFSM4VMZF3HQ> . You are receiving this because you were mentioned.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1001/comments/3053648@ github.com>

[ColinKennedy]

Yeah it was. I probably should've wrote that when I wasn't so sleepy, whoops! Anyway though, all of my original points still stand (I'll edit the post though :P)