Python package variants for different operating systems

[hughetop]

Branching from a different discussion, I'm curious how to handle python packages for multiple operating systems. The rez-bind python command fails on windows due to permissions errors, so I had to hand-edit the package.py to have something like this:

def commands():
    import sys
    if sys.platform == 'win32':
        env.PATH.prepend(r'C:\Program Files\Python39')
    elif sys.platform == 'darwin':
        env.PATH.prepend('/Library/Frameworks/Python.framework/Versions/3.9/bin')

That way when python-3.9 is included in a rez-environment, the correct python executable is added to the path. I know this isn't how it should be done and I should create variants for each platform. The problem with that is the permissions error makes this difficult/impossible. So what's the correct way to handle the python package for multiple platforms?

As a side note, it's the same with any 3rd party software. It gets installed locally (for windows and mac), so I don't understand how using platform-specific variants will solve the problem. All the package.py can do is update the PATH environment variable appropriately depending on the operating system.

[instinct-vfx]

Disclaimer: Windows shop here, so rez experience on windows is by FAR my better side. So take with a pinch of salt for other platforms.

Multi-platform packages are typically handled with variants. See the Docs on Variants here: https://github.com/AcademySoftwareFoundation/rez/wiki/Variants

You would still be making the same distinction and it would be in package.py but rez would know about variants being present and also reflect that in resolves rather than magically handle it in the package using sys.

Sepcifically for python I am a strong proponent of having full python rez packages rather then packages that just wrap around local installations. It fails very easily (e.g. when the local installation is a different patch version, or no installation is present at all).

Specifically for windows I had a little live session with the TAO crowd to build a small tool to automatically release python releases as rez packages. While there is no recording of the session, i made the code available here: https://github.com/techartorg/rez_utils/tree/main/rezify_python

Beware, i would not really consider this production proven or prime time ready, but it gives a lot of hints on how to do something like that. Just to repeat: YMMV on Linux :)

[hughetop]

I guess I still don't understand how variants help in this situation. I know how they're used for internally developed packages, or pip-installed packages, but not 3rd party software. Using an example similar to the docs:

/rez/packages/python/3.9.10/platform-windows/<PAYLOAD>
                           /platform-osx/<PAYLOAD>
                           package.py

I don't know what the actual payload would be here. Python is installed locally, not to the network, so the payload would be empty. It's the same with a 3rd party DCC, installed locally, so the rez package is just a thin wrapper around local installs that adds environment variables.

I've played around a bit with a "portable" python installed to the network, but it was so prohibitively slow on windows (or any home computers connected with VPN rather than on-site computers) that it wasn't worth it. I'll take a look at rezify_python though.

Ironically, I won't need to support windows in the future, just linux and mac os. But since osx requires local software installs, it's still relevant for me.

[hughetop]

And out of curiosity, what shell do you use on windows? I tried to use git-bash for a long time, but it just didn't play well with rez, so I switched to Cmder, which is more like an extension for cmd/powershell.

[instinct-vfx]

Trying to adress your points in order:

It does not really matter if you are doing internal packages or pip or locally installed packages. Variants are the concept in rez to handle different variations of the same package version. In this case, different platforms. For pip packages, that might extend to different python versions. And for Maya Plugins that might also include different host versions. Adding variants does not only create subfolders for the payloads (no matter if you use them or not) but more importantly it makes the variants known to rez and the solver. A very simple (and by far not the only!) case where this helps is as follows:

Lets use your example there. You want to support windows and mac initially. So you add the handling of platform specifics to your package.py without variants. Now, a while later you add linux clients. Suddenly all your tools and shells miserably fail on linux because linux might require yet another different path for python. And noone knows why.
If you do the same with variants, your resolves will fail with an error message and a failgraph to tell you that there is no linux variant of your python package available and you need to provide one.

We handle all platform specifics as variants, and that's also how rez is designed, and even if it seems trivial at first to not do that, there is little benefit to not doing it, but a lot of potential errors down the road. Variants are very useful independently of having a payload or not.

As for portable python: I would suggest to look into rez's payload caching. It caches package payloads locally (note: It's important that it only caches payloads and NOT the actual packages as a whole) and removes these issues. It is very nicely designed and supports automatic or semi-automatic clean-up as well. See https://github.com/AcademySoftwareFoundation/rez/wiki/Managing-Packages#package-caching for more information.

In regard to the shell on windows my recommendation would be Powershell (or more specifically pwsh if you can). It does some things a lot better than the cmd implementation.

[hughetop]

Ah, I see, that makes a lot of sense regarding the resolve error message. So if I utilize variants and add those payload subdirectories (even if they're empty), it would make it much easier to debug resolve issues.

Payload caching is one of the best features of rez in my opinion, especially on windows. But I ultimately decided against using it for 3rd party software and DCCs. For DCCs, they still need local installs, I couldn't find a way around that. For python, it didn't make much sense to me since rez is secretly just python with exe wrappers, and we needed to have python installed locally anyway. But since Linux and osx have built-in python executables, I'll probably try to have rez use those, and deploy other versions of python as rez packages and cache them.

This clears up a lot of my confusion. Thank you!