Improving rez.vendor embedded requirements

[ttrently]

I'd like to propose some updates to how rez.vendor packages are handled and how we might move away from embedded requirements.

Recently we have had the need with internal development to include additional packages into the rez/vendor location, but have realized there's a hidden implementation cost. This cost is the need to correct the packages internal imports to be from absolute to relative in order to function correctly from rez.vendor.

Take for example the pymongo package, by default when installed, this package references all of its imports as absolute, but makes it unusable from within rez.vendor.

from pymongo import _csot

Normally to correct this, once the package is installed to the rez/vendor location, an additional refactor pass must be made to correct these import statements and tested, some of which can be automated, but can require some manual work as well depending on the size of the install.

In this case, the previous import line would need to be changed to:

from . import _csot # OR
from rez.vendor.pymongo import _csot

This can become tedious though as more requirements are added and managed.

Proposal

site-packages

Treat rez requirements as such through an internal site-packages directory that is bootstrapped at runtime or when a requested plugin is loaded.

Example:

We have a package_repository for mongo that has a requirement on the pymongo package. Normally we would either manually install it to rez.vendor or to the global rez install location. The only problem with this is time taken to maintain the requirement and that it is not recognized when binding or installing rez as a package.

The solution here would be to install pymongo as a requirement of the mongo package repository under site-packages and have the initialization add the site location when requested.

package_repository/
├── site-packages/
│   ├── bson
│   ├── dns
│   ├── gridfs
│   └── pymongo
├── __init__.py
├── mongo.py
└── filesystem.py

Then in __init__.py of the package repository plugin:

from rez.plugin_managers import extend_path, extend_site
__path__ = extend_path(__path__, __name__)
extend_site(__file__) # calls site.addsitedir('__file__/site-packages')

This can then be implemented at higher levels as well from within rez, replacing rez.vendor with a rez/site-packages location that is added upon rez initialization. The few benefits of this:

• Pythonic and standard way of handling additional packages.
• Compact and only impacts the rez environment and not a users default Python install.
• Can take advantage of requirements.txt to handle installs either during a rez install or when distributing specific plugins.

I'd be interested in others thoughts or if there may be another way to handle this. In our case we were looking for a low-effort, minimum-impact solution to providing additional requirements without having to either manipulate the installer, or incur the work of converting these libraries.

[JeanChristopheMorinPerso]

Hi @ttrently , thanks for bring the subject of our vendored dependencies here.

I think the way forward would be to unvendor everything. As you said it, vendoring has a huge cost and to me only has downsides. And yes I really think there is no upside to vendoring in rez.

Vendoring dependencies makes it pretty painful to upgrade dependencies, it's a security risk that I personally don't want to deal with, it makes it more difficult (or even impossible) to track where the code comes from (again a security issue).

For your specific case, I would suggest that the S3 repository be a plugin, external to the main rez repo and make it installable via pip. This would allow you to have the dependencies that you want without any downsides.

One thing to note is that the TSC members slightly touched that subject not too long ago, but we unfortunately didn't go far in our conversation. From what I remember we mostly stated where we stand in regards to vendored dependencies. The result was mixed.

(I already anticipate replies saying that rez should be self contained because some companies/studios block internet from machines. And to me that is not a good argument to vendor dependencies. If a studio/company wants to block internet access, that is their problem, not our. These companies/studios already know how to deal with that and already have processes and the infrastructure in place to proxy package indexes (PyPI, etc)).

This subject will be like a pandora box though. It touches the installer, dependencies (obviously), #1349, security, how plugins are implemented and deployed, etc.
So on in all, I'd rather spend energy on unvendoring than keeping the vendored dependencies and have to live with the fact that we could have a mongo DB client library that in the future could have security vulnerabilities, in which case I don't know if anybody would every be made aware of because it would be vendored and so made "invisible".

One of the only case I can think of where vendoring makes sense is for bootstrapping purposes. For example pip vendors its dependencies because for them it has more advantages than disadvantages. But in our case, bootstrapping shouldn't (and isn't) an issue.

[ttrently]

Thanks for all the insight here @JeanChristopheMorinPerso!

Like you had mentioned, I think we are going to go that route for now to make things easy, in that we will make our Mongo and AWS plugins external from the main rez repo and make it installable via pip. This should hopefully cut down on work in the future internally with at least managing additional plugins and their requirements.

I'm most likely in the same line of thought though with unvendoring rez and treating those dependencies as such, but like you had mentioned I can only imagine what that does for installers, security, and plugin implementation.

[JeanChristopheMorinPerso]

Closing in favor of #1668.