Beta packages

[jaredauty]

I'm getting a little stuck on the best way to handle testing packages safely. Often I have cases where I need new versions of packages to be available to the whole studio (i.e. used on the farm or given to users to test) but I don't want them getting picked up in production environments.

For example I might have a production environment that requires
foo-1.2

Lets say foo-1.2 requires bar-2.4+

Now I want to release a version of bar (2.5.0) for wider testing, but I don't want it picking up in any production environments.

Ideally I'd like to release bar-2.5.0.beta.1 (to indicate that it's the first beta version of 2.5.0), I could then use a filter to make sure anything with "beta" is not allowed during resolve.

This leaves the issue of how do I tell the beta environment to allow my bar-2.5.0.beta.1 in? I don't want to turn off the filter because I don't want any other beta versions of other packages getting in. I just want my bar package in beta.

My ideal solution would be to have a filter that will filter out anything with a beta unless it's explicitly declared in the context requirements. That way I could set my test environment to foo-1.2 bar-2.5.0.beta and it will know that it should allow the beta for that package.

I've managed to make that work by running the resolves using the python API and subclassing the filter that I pass into ResolvedContext. This works but ideally I'd like to have a solution that works from the rezconfig.py so the behavior is consistent across all the rez commands.

Is there a way to create custom filters inside the rezconfig.py and if so can it have access to the context requirements?
Is there a better way to do this?
Am I thinking about this wrongly somehow?

Thanks

[instinct-vfx]

There are many different ways to achieve what you are after and also different approaches in general to introducing non production packages. Here is a rough high level overview of what we do:

We have multiple package repositories that reflect the different stages in the lifecycle of a package: dev, staging, prod, deprecate, archive. These have also specific permissions (so dev is only available for devs, staging for devs+testers prod for everyone else etc.). Depending on the desired environment we assemble different packages_path chains. So for example a dev environment would source dev->staging->prod->deprecate whereas a prod environment would only source prod->deprecate as an example.

As these repository are additive and first takes precedence even with the same version you can create most of the scenarios described above. That said however, a setup as specific as the one you described above would become tricky quickly depending on your size, the amount of devs, packages and releases i guess.

With all that said, i wonder if what you do might be possible with package filter and an override. You could have a standard filter that excludes all packages containing beta and then override that to a filter that excludes all packages with beta but includes bar. The wiki has such an example:

package_filter:
    excludes:
    - glob(*.beta)
    includes:
    - glob(foo-*)

[jaredauty]

Hey @instinct-vfx
That makes a lot of sense. I like the idea of different repos. I've got a couple of questions about how you manage them.

1. Do you copy the packages from one repo to the next or move or release with different version scheme?
2. How does the deprecate repo work? I presume things get moved in there when they're no longer needed but if they're referenced in your production env what does this achieve?

Thanks

[instinct-vfx]

As for 1) Both works i would say and it depends on your setup and requirements. We do releases, but not with a different versioning scheme or versions. If you go the move route, make sure you use rez-mv (https://github.com/nerdvegas/rez/wiki/Command-Line-Tools#rez-mv)
For 2) The idea here is to be able to warn/and escalate. The logic is: When we want to decommission something it goes to deprecate first. To really decommission it would go to the archive repo (which no one has access to). Before moving to archive we can then check using the stats we get from the AMQP integration to make sure no one is still using the packages, or at least give them a heads up.

We are not really practicing the decomm workflow atm as there is little to decomm, but that's the idea.

[nerdvegas]

The simplest way to do this is: rez-env eek bah-2.5.beta --include bah However, there's currently no way to do what you're asking for specifically (ie automate this behaviour for all packages). Supporting this is problematic. Everything you pass to rez-env is a _request_, not a specific package version (even if you specify bah==2.5.0.beta, this is just a very narrow version range request on package bah). So how do you even define this behaviour? If we say, "apply package filters, but also allow any package within a request in the initial list", this doesn't help us - "rez-env bah" would then include all beta versions of bah. Whilst it's intuitive to us humans to see a request for bah-2.5.0.beta and to see that we do want beta versions of bah, there isn't a generic way to express that which would work for all filters (and if you try, you end up with ambiguous cases. What if you had a simple range exclusion filter set on foo-1, but then a "rez-env foo-1.2" caused that to be ignored?) The only option I think, would be to always include packages requested with == syntax... but that doesn't feel right to me. Thoughts welcome A

…

On Mon, Jun 7, 2021 at 5:22 PM Thorsten Kaufmann ***@***.***> wrote: There are many different ways to achieve what you are after and also different approaches in general to introducing non production packages. Here is a rough high level overview of what we do: We have multiple package repositories that reflect the different stages in the lifecycle of a package: dev, staging, prod, deprecate, archive. These have also specific permissions (so dev is only available for devs, staging for devs+testers prod for everyone else etc.). Depending on the desired environment we assemble different packages_path chains. So for example a dev environment would source dev->staging->prod->deprecate whereas a prod environment would only source prod->deprecate as an example. As these repository are additive and first takes precedence even with the same version you can create most of the scenarios described above. That said however, a setup as specific as the one you described above would become tricky quickly depending on your size, the amount of devs, packages and releases i guess. With all that said, i wonder if what you do might be possible with package filter and an override. You could have a standard filter that excludes all packages containing beta and then override that to a filter that excludes all packages with beta but includes bar. The wiki has such an example: package_filter: excludes: - glob(*.beta) includes: - glob(foo-*) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1093 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSSQKWZBSQAVW3S4SB3TRRXTJANCNFSM46CO5EYA> .

[jaredauty]

Ok, thanks for the explanation. Leads me to think I'm looking in the wrong direction, very good to know.