How to handle python 2 and 3 together? #1016
Unanswered
anderslanglands
asked this question in
Q&A
Replies: 11 comments 4 replies
-
|
Hey Anders, Step 1, as I would put it, is to build python-2 and 3 packages. To do so, we have a simple "gcc" package version which is keyed to our operating systems' gcc, and does little else but make sure that CC/CXX/PATH are set, etc. From there, we've got some minimal cmake which will build python to a package root, and a package.py that will expose it's tool names, private_build_require the gcc and cmake packages, etc. So we build both of those. (In our case, we're currently maintaining variants against 2.7, 3.6, 3.7, 3.8, and will add 3.9 as we drop 2.7 as other dependencies allow). The variants section of a package that is varianting against python-2 and 3, might look like this: And that would build variants against the latest version of each of those in your repository. And that would build all three of those. Versions are somewhat random based on our timing of building those packages. Let me know if that helps, or, more specifically, if that does not, or leaves a lot of open questions. It is a tiny bit hard to tell what you are looking for, specifically, like, if you need help with the building phases, or if your existing python packages are just providing your OS-level python or whatever. A facility can make a lot of different choices that affect how this works, like, maybe pip-installing a pile of stuff at the OS level and not packaging them all up, which is the rather zealous approach that we take. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks for the detailed response! In my case I'm doing this on my personal development machine(s) and I just want to be able to start building libraries against both python2 and python3 as the vfx world migrates. So really I just want to wrap the system python. My first stumbling block is what a python3 package.py would look like. Something like this? and then I'd just add to all the package.py's of the libraries when I rez-build them and everything should just work? Cheers, |
Beta Was this translation helpful? Give feedback.
-
|
Heya Anders,
I'd advise against pointing to the system interpreter...
I have a python rez package you're welcome to use:
https://gist.github.com/zachlewis/d7bee5483b96181631fafffe6bff8bbc
(there's a TON going on in that package.py! I should probably annotate
it...)
The package should "just work" for any python version -- just change the
version string to whichever python you'd prefer, and rez-build.
(tested on macos / centos / ubuntu)
Note in the requirements, you'll need packages for gcc, cmake, and zlib, so
build or bind these first:
gcc: (just run `$ rez-bind gcc`)
cmake: https://gist.github.com/zachlewis/b7e6b85cb3d1b948405c5a0061533db9
(or bind to system cmake with `$ rez-bind cmake`)
zlib: https://gist.github.com/zachlewis/6a87445194e67d760da7f1c5589679f9
(or use the system zlib and just comment out the package requirement)
Alternatively, you can "bind" your system python interpreter (whichever was
used when installing rez) with the `rez-bind python` command; but that will
only get you either python-2 or python-3, and not both.
And yes, for building packages against python-2 and python-3 variants,
specifying `variants = [['python-2'], ['python-3']]` should work, although
if there's a chance you'll have more than one version of python 2.x and 3.x
at some point in the future, you might want to specify minor versions, just
to be safe.
Let me know if you're having trouble...
Z
P.S. Loved that PhysLight talk!
…On Sat, Jan 23, 2021 at 8:12 PM Anders Langlands ***@***.***> wrote:
Thanks for the detailed response!
In my case I'm doing this on my personal development machine(s) and I just
want to be able to start building libraries against both python2 and
python3 as the vfx world migrates. So really I just want to wrap the system
python.
My first stumbling block is what a python3 package.py would look like.
Something like this?
name = "python"
version = "3.6.12"
tools = ["python3"]
variants = [["platform-linux", "arch-x86_64", "os-Ubuntu-18.04"]]
def commands():
env.PYTHON_EXECUTABLE='/usr/bin/python3'
env.Python_EXECUTABLE='/usr/bin/python3'
and then I'd just add
variants = [['python-2'], ['python-3']
to all the package.py's of the libraries when I rez-build them and
everything should just work?
Cheers,
Anders
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1016 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARAC4B64PNI276T7WJ5LHLS3NXXRANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Hi Zach, Any change that you have been keeping these up to date and just have not updated the gists? |
Beta Was this translation helpful? Give feedback.
-
|
I ended up getting python working... a careful sprinkling of |
Beta Was this translation helpful? Give feedback.
-
|
I'm facing some of a similar issue. What am I doing wrong exactly? Should I omit binding both interpreters using Thanks! |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I've never gotten that `rez-bind python --exe` thing to work properly, to
be honest; and I kind of feel the `--exe` option should be removed.
(Specifically, that CodecRegistryError occurs when you install rez with
python-2, and try to rez-bind an external python-3)
`rez-bind` itself is kind of a hack to get up and running with rez quickly
-- it solves a chicken-and-egg issue with trying to build, for instance,
gcc without first having a rez package for gcc. As such, it's provided as a
convenience for a handful of tools and libraries, but binding rez package
contents to libraries and tools managed outside of rez kind of goes against
rez's ethos.
Perhaps @nerdvegas can elaborate; my sense is, rez-bind is kind of
reluctantly supported, and its use isn't really encouraged if alternatives
are available.
So, my advice is to definitely bite the bullet and create a proper python-3
package.
The gist I posted above should work well -- it's a little gnarly-looking,
perhaps a little rough around the edges, but it's informed by a good deal
of experimentation and best practices used in our production python
packages at work (full disclosure, I created this package for my own home
use; this isn't exactly what we use in production). Notably, it installs
(and updates) pip and setuptools along with python, which is strongly
recommended, and lets you use `rez-pip` out of the box. It also goes
through some trouble to make python-2 and python-3 packages behave the same
way (i.e., for python3, it symlinks "python", "pip", etc, to "python3",
"pip3", etc., which is super handy for deploying custom scripts that begin
with `!#/usr/bin/env python`).
I also have a marginally cleaner, ever-so-slightly different version of my
python package here:
https://gist.github.com/zachlewis/8e0b6b0bb089f45b9df7ac85904dcfbd
(I prefer this package definition over the one I shared earlier, but
someone was having trouble with this one, although I never found what
trouble they were having, exactly... would be curious to know if others
have trouble. I will also note that I usually build against additional 'xz'
and sometimes 'libexpat' packages, for native support for tar compression
stuff and xml stuff; and I usually don't bother with 'zlib', unless there's
some reason to believe the system zlib isn't available).
There are also a handful of other python-3 package definitions floating
around github, if you have a look... IIRC, @predat has a platform-agnostic
python-3.6 installer in their rez-repository repo; and I just stumbled upon
https://github.com/davidlatwe/rez-miniconda which I'm really curious to try
out (although personally, I've only ever used CPython interpreters with
rez). Allan also has an example python package in his rez-recipes repo, but
it's a little ancient...
In a perfect world, someone might want to look into hacking [pyenv](
https://github.com/pyenv/pyenv) to build and deploy pythons for rez...
…On Mon, Jan 25, 2021 at 3:54 AM Alberto Sierra ***@***.***> wrote:
I'm facing some of a similar issue.
I'm trying to "bind" both interpreters, python2 and python3. I originally
installed Rez using the python2 interpreter. When I try to bind the python3
interpreter using Rez-bind python --exe which python3, It creates the
corresponding package, but when I try to call python within that new
environment, I get the following Python error:
Fatal Python error: initfsencoding: unable to load the file system codec
File "/Users/albertosierra/packages/python/3.7.7/platform-osx/arch-x86_64/os-osx-10.16/python/lib/encodings/__init__.py", line 123
raise CodecRegistryError,\
^
SyntaxError: invalid syntax
Current thread 0x000000010d0bbe00 (most recent call first):
[1] 13184 abort python
What am I doing wrong exactly? Should I omit binding both interpreters
using Rez-bind and create my own python3 package manually?
Thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1016 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARAC4F6T6ADUCWUQK2QVNTS3UWUFANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hey all, a couple of things to mention on this.
First - rez-bind should be deprecated, since you can (for a long time now)
achieve all the same things with native package.py files.
For example, see
https://github.com/nerdvegas/rez-recipes/blob/master/python/_native/package.py.
I wouldn't consider this a working package, but you can use it as a guide.
You can see how it introspects the system at build time to generate the
correct package attributes.
Second - rez does need a way to express the idea of "provides". With this
feature, a "provided" package implies that the package already exists
(either on the system, or via another package) and its effect should be a
no-op in the runtime. I have plans to support this using the new ephemerals
feature that was released a few weeks ago. Some examples of what it might
look like:
```
# set an implicit that tells rez that python-2.7.5 is already available, so
don't bind it (ie don't consumes its matching rez pkg (if there is one) and
don't run its pkg commands)
]$ export
REZ_IMPLICIT_PACKAGES="~platform==linux:~arch==x86_64:~os==Ubuntu-16.04:.provided.python-2.7.5"
```
Many DCCs include their own embedded python, they could use the same
feature to stop rez configuring its own python pkg (which tbh is why this
would be implemented as an ephemeral in the first place):
```
# in maya package.py
requires = [
...,
.provides.python-2.7.5
]
```
For convenience, I'd probably also include this form (which would be
equivalent to the first example):
```
]$ REZ_PROVIDED=python-2.7.5 rez-env ...
```
Links:
https://github.com/nerdvegas/rez/wiki/Ephemeral-Packages
Hth
A
…On Tue, Jan 26, 2021 at 4:47 AM zachlewis ***@***.***> wrote:
I've never gotten that `rez-bind python --exe` thing to work properly, to
be honest; and I kind of feel the `--exe` option should be removed.
(Specifically, that CodecRegistryError occurs when you install rez with
python-2, and try to rez-bind an external python-3)
`rez-bind` itself is kind of a hack to get up and running with rez quickly
-- it solves a chicken-and-egg issue with trying to build, for instance,
gcc without first having a rez package for gcc. As such, it's provided as a
convenience for a handful of tools and libraries, but binding rez package
contents to libraries and tools managed outside of rez kind of goes against
rez's ethos.
Perhaps @nerdvegas can elaborate; my sense is, rez-bind is kind of
reluctantly supported, and its use isn't really encouraged if alternatives
are available.
So, my advice is to definitely bite the bullet and create a proper python-3
package.
The gist I posted above should work well -- it's a little gnarly-looking,
perhaps a little rough around the edges, but it's informed by a good deal
of experimentation and best practices used in our production python
packages at work (full disclosure, I created this package for my own home
use; this isn't exactly what we use in production). Notably, it installs
(and updates) pip and setuptools along with python, which is strongly
recommended, and lets you use `rez-pip` out of the box. It also goes
through some trouble to make python-2 and python-3 packages behave the same
way (i.e., for python3, it symlinks "python", "pip", etc, to "python3",
"pip3", etc., which is super handy for deploying custom scripts that begin
with `!#/usr/bin/env python`).
I also have a marginally cleaner, ever-so-slightly different version of my
python package here:
https://gist.github.com/zachlewis/8e0b6b0bb089f45b9df7ac85904dcfbd
(I prefer this package definition over the one I shared earlier, but
someone was having trouble with this one, although I never found what
trouble they were having, exactly... would be curious to know if others
have trouble. I will also note that I usually build against additional 'xz'
and sometimes 'libexpat' packages, for native support for tar compression
stuff and xml stuff; and I usually don't bother with 'zlib', unless there's
some reason to believe the system zlib isn't available).
There are also a handful of other python-3 package definitions floating
around github, if you have a look... IIRC, @predat has a platform-agnostic
python-3.6 installer in their rez-repository repo; and I just stumbled upon
https://github.com/davidlatwe/rez-miniconda which I'm really curious to
try
out (although personally, I've only ever used CPython interpreters with
rez). Allan also has an example python package in his rez-recipes repo, but
it's a little ancient...
In a perfect world, someone might want to look into hacking [pyenv](
https://github.com/pyenv/pyenv) to build and deploy pythons for rez...
On Mon, Jan 25, 2021 at 3:54 AM Alberto Sierra ***@***.***>
wrote:
> I'm facing some of a similar issue.
> I'm trying to "bind" both interpreters, python2 and python3. I originally
> installed Rez using the python2 interpreter. When I try to bind the
python3
> interpreter using Rez-bind python --exe which python3, It creates the
> corresponding package, but when I try to call python within that new
> environment, I get the following Python error:
>
> Fatal Python error: initfsencoding: unable to load the file system codec
> File
"/Users/albertosierra/packages/python/3.7.7/platform-osx/arch-x86_64/os-osx-10.16/python/lib/encodings/__init__.py",
line 123
> raise CodecRegistryError,\
> ^
> SyntaxError: invalid syntax
>
> Current thread 0x000000010d0bbe00 (most recent call first):
> [1] 13184 abort python
>
> What am I doing wrong exactly? Should I omit binding both interpreters
> using Rez-bind and create my own python3 package manually?
>
> Thanks!
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <
#1016 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AARAC4F6T6ADUCWUQK2QVNTS3UWUFANCNFSM4WPJAWDA
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1016 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSV4DWLGTJPPZLLLRGDS3WVDNANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks Allan --
That's really interesting.
So, I guess one would have to be mindful of upsetting expectations of
downstream packages whose commands rely on upstream package rez semantics.
For instance...
```
# in OpenColorIO package.py
def pre_build_commands():
env.REZ_BUILD_CMAKE_OPTIONS = " ".join([
...,
"-DPython_EXECUTABLE={resolve.python.root}/bin/python",
])
```
Presumably, if we wanted to build a "maya" variant, and the "maya" package
"provides" python, then "python" would no longer be in the resolve (even if
explicitly requested).
I suppose in practice that just means adding a simple ["maya"] variant,
alongside various python version variants, and adjusting commands to
control for ` if "maya" in resolve`; i.e.,
```
# in adjusted OpenColorIO package.py
def pre_build_commands():
if 'maya' in resolve:
python_exe = '{resolve.maya.root}/path/to/local/python'
else:
python_exe = '{resolve.python.root}/bin/python'
env.REZ_BUILD_CMAKE_OPTIONS = " ".join([
...,
"-DPython_EXECUTABLE={python_exe}"
])
```
Does that sound about right?
On Tue, Jan 26, 2021 at 5:23 PM allan johns <[email protected]>
wrote:
… Hey all, a couple of things to mention on this.
First - rez-bind should be deprecated, since you can (for a long time now)
achieve all the same things with native package.py files.
For example, see
https://github.com/nerdvegas/rez-recipes/blob/master/python/_native/package.py
.
I wouldn't consider this a working package, but you can use it as a guide.
You can see how it introspects the system at build time to generate the
correct package attributes.
Second - rez does need a way to express the idea of "provides". With this
feature, a "provided" package implies that the package already exists
(either on the system, or via another package) and its effect should be a
no-op in the runtime. I have plans to support this using the new ephemerals
feature that was released a few weeks ago. Some examples of what it might
look like:
```
# set an implicit that tells rez that python-2.7.5 is already available, so
don't bind it (ie don't consumes its matching rez pkg (if there is one) and
don't run its pkg commands)
]$ export
REZ_IMPLICIT_PACKAGES="~platform==linux:~arch==x86_64:~os==Ubuntu-16.04:.provided.python-2.7.5"
```
Many DCCs include their own embedded python, they could use the same
feature to stop rez configuring its own python pkg (which tbh is why this
would be implemented as an ephemeral in the first place):
```
# in maya package.py
requires = [
...,
.provides.python-2.7.5
]
```
For convenience, I'd probably also include this form (which would be
equivalent to the first example):
```
]$ REZ_PROVIDED=python-2.7.5 rez-env ...
```
Links:
https://github.com/nerdvegas/rez/wiki/Ephemeral-Packages
Hth
A
On Tue, Jan 26, 2021 at 4:47 AM zachlewis ***@***.***>
wrote:
> I've never gotten that `rez-bind python --exe` thing to work properly, to
> be honest; and I kind of feel the `--exe` option should be removed.
> (Specifically, that CodecRegistryError occurs when you install rez with
> python-2, and try to rez-bind an external python-3)
>
> `rez-bind` itself is kind of a hack to get up and running with rez
quickly
> -- it solves a chicken-and-egg issue with trying to build, for instance,
> gcc without first having a rez package for gcc. As such, it's provided
as a
> convenience for a handful of tools and libraries, but binding rez package
> contents to libraries and tools managed outside of rez kind of goes
against
> rez's ethos.
>
> Perhaps @nerdvegas can elaborate; my sense is, rez-bind is kind of
> reluctantly supported, and its use isn't really encouraged if
alternatives
> are available.
>
> So, my advice is to definitely bite the bullet and create a proper
python-3
> package.
>
> The gist I posted above should work well -- it's a little gnarly-looking,
> perhaps a little rough around the edges, but it's informed by a good deal
> of experimentation and best practices used in our production python
> packages at work (full disclosure, I created this package for my own home
> use; this isn't exactly what we use in production). Notably, it installs
> (and updates) pip and setuptools along with python, which is strongly
> recommended, and lets you use `rez-pip` out of the box. It also goes
> through some trouble to make python-2 and python-3 packages behave the
same
> way (i.e., for python3, it symlinks "python", "pip", etc, to "python3",
> "pip3", etc., which is super handy for deploying custom scripts that
begin
> with `!#/usr/bin/env python`).
>
> I also have a marginally cleaner, ever-so-slightly different version of
my
> python package here:
> https://gist.github.com/zachlewis/8e0b6b0bb089f45b9df7ac85904dcfbd
> (I prefer this package definition over the one I shared earlier, but
> someone was having trouble with this one, although I never found what
> trouble they were having, exactly... would be curious to know if others
> have trouble. I will also note that I usually build against additional
'xz'
> and sometimes 'libexpat' packages, for native support for tar compression
> stuff and xml stuff; and I usually don't bother with 'zlib', unless
there's
> some reason to believe the system zlib isn't available).
>
> There are also a handful of other python-3 package definitions floating
> around github, if you have a look... IIRC, @predat has a
platform-agnostic
> python-3.6 installer in their rez-repository repo; and I just stumbled
upon
> https://github.com/davidlatwe/rez-miniconda which I'm really curious to
> try
> out (although personally, I've only ever used CPython interpreters with
> rez). Allan also has an example python package in his rez-recipes repo,
but
> it's a little ancient...
>
> In a perfect world, someone might want to look into hacking [pyenv](
> https://github.com/pyenv/pyenv) to build and deploy pythons for rez...
>
>
>
> On Mon, Jan 25, 2021 at 3:54 AM Alberto Sierra ***@***.***
>
> wrote:
>
> > I'm facing some of a similar issue.
> > I'm trying to "bind" both interpreters, python2 and python3. I
originally
> > installed Rez using the python2 interpreter. When I try to bind the
> python3
> > interpreter using Rez-bind python --exe which python3, It creates the
> > corresponding package, but when I try to call python within that new
> > environment, I get the following Python error:
> >
> > Fatal Python error: initfsencoding: unable to load the file system
codec
> > File
>
"/Users/albertosierra/packages/python/3.7.7/platform-osx/arch-x86_64/os-osx-10.16/python/lib/encodings/__init__.py",
> line 123
> > raise CodecRegistryError,\
> > ^
> > SyntaxError: invalid syntax
> >
> > Current thread 0x000000010d0bbe00 (most recent call first):
> > [1] 13184 abort python
> >
> > What am I doing wrong exactly? Should I omit binding both interpreters
> > using Rez-bind and create my own python3 package manually?
> >
> > Thanks!
> >
> > —
> > You are receiving this because you commented.
> > Reply to this email directly, view it on GitHub
> > <
>
#1016 (comment)
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AARAC4F6T6ADUCWUQK2QVNTS3UWUFANCNFSM4WPJAWDA
> >
> > .
> >
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#1016 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAMOUSV4DWLGTJPPZLLLRGDS3WVDNANCNFSM4WPJAWDA
>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1016 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARAC4D5AAH4H2ZMQACXNTLS346G5ANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Something approximating that I think yeah.Or perhaps we add a new
`provided` builtin like so:
```
provider = provided('python') # return either Variant that provides, or
True if provided by system
if provider is True:
python_exe = '/usr/bin/python'
elif provider:
python_exe = provider.root + '/path/to/local/python'
else:
python_exe = '{resolve.python.root}/bin/python'
```
I _ think_ you could improve this further by adding a custom 'python_exe'
attrib to relevant packages too:
```
if provider is True:
python_exe = '/usr/bin/python'
else:
pkg = provider or resolve.python
python_exe = pkg.python_exe
```
…On Thu, Jan 28, 2021 at 1:51 AM zachlewis ***@***.***> wrote:
Thanks Allan --
That's really interesting.
So, I guess one would have to be mindful of upsetting expectations of
downstream packages whose commands rely on upstream package rez semantics.
For instance...
```
# in OpenColorIO package.py
def pre_build_commands():
env.REZ_BUILD_CMAKE_OPTIONS = " ".join([
...,
"-DPython_EXECUTABLE={resolve.python.root}/bin/python",
])
```
Presumably, if we wanted to build a "maya" variant, and the "maya" package
"provides" python, then "python" would no longer be in the resolve (even if
explicitly requested).
I suppose in practice that just means adding a simple ["maya"] variant,
alongside various python version variants, and adjusting commands to
control for ` if "maya" in resolve`; i.e.,
```
# in adjusted OpenColorIO package.py
def pre_build_commands():
if 'maya' in resolve:
python_exe = '{resolve.maya.root}/path/to/local/python'
else:
python_exe = '{resolve.python.root}/bin/python'
env.REZ_BUILD_CMAKE_OPTIONS = " ".join([
...,
"-DPython_EXECUTABLE={python_exe}"
])
```
Does that sound about right?
On Tue, Jan 26, 2021 at 5:23 PM allan johns ***@***.***>
wrote:
> Hey all, a couple of things to mention on this.
>
> First - rez-bind should be deprecated, since you can (for a long time
now)
> achieve all the same things with native package.py files.
> For example, see
>
>
https://github.com/nerdvegas/rez-recipes/blob/master/python/_native/package.py
> .
> I wouldn't consider this a working package, but you can use it as a
guide.
> You can see how it introspects the system at build time to generate the
> correct package attributes.
>
> Second - rez does need a way to express the idea of "provides". With this
> feature, a "provided" package implies that the package already exists
> (either on the system, or via another package) and its effect should be a
> no-op in the runtime. I have plans to support this using the new
ephemerals
> feature that was released a few weeks ago. Some examples of what it might
> look like:
>
> ```
> # set an implicit that tells rez that python-2.7.5 is already available,
so
> don't bind it (ie don't consumes its matching rez pkg (if there is one)
and
> don't run its pkg commands)
> ]$ export
>
>
REZ_IMPLICIT_PACKAGES="~platform==linux:~arch==x86_64:~os==Ubuntu-16.04:.provided.python-2.7.5"
> ```
>
> Many DCCs include their own embedded python, they could use the same
> feature to stop rez configuring its own python pkg (which tbh is why this
> would be implemented as an ephemeral in the first place):
>
> ```
> # in maya package.py
> requires = [
> ...,
> .provides.python-2.7.5
> ]
> ```
>
> For convenience, I'd probably also include this form (which would be
> equivalent to the first example):
>
> ```
> ]$ REZ_PROVIDED=python-2.7.5 rez-env ...
> ```
>
> Links:
> https://github.com/nerdvegas/rez/wiki/Ephemeral-Packages
>
> Hth
> A
>
>
>
>
>
> On Tue, Jan 26, 2021 at 4:47 AM zachlewis ***@***.***>
> wrote:
>
> > I've never gotten that `rez-bind python --exe` thing to work properly,
to
> > be honest; and I kind of feel the `--exe` option should be removed.
> > (Specifically, that CodecRegistryError occurs when you install rez with
> > python-2, and try to rez-bind an external python-3)
> >
> > `rez-bind` itself is kind of a hack to get up and running with rez
> quickly
> > -- it solves a chicken-and-egg issue with trying to build, for
instance,
> > gcc without first having a rez package for gcc. As such, it's provided
> as a
> > convenience for a handful of tools and libraries, but binding rez
package
> > contents to libraries and tools managed outside of rez kind of goes
> against
> > rez's ethos.
> >
> > Perhaps @nerdvegas can elaborate; my sense is, rez-bind is kind of
> > reluctantly supported, and its use isn't really encouraged if
> alternatives
> > are available.
> >
> > So, my advice is to definitely bite the bullet and create a proper
> python-3
> > package.
> >
> > The gist I posted above should work well -- it's a little
gnarly-looking,
> > perhaps a little rough around the edges, but it's informed by a good
deal
> > of experimentation and best practices used in our production python
> > packages at work (full disclosure, I created this package for my own
home
> > use; this isn't exactly what we use in production). Notably, it
installs
> > (and updates) pip and setuptools along with python, which is strongly
> > recommended, and lets you use `rez-pip` out of the box. It also goes
> > through some trouble to make python-2 and python-3 packages behave the
> same
> > way (i.e., for python3, it symlinks "python", "pip", etc, to "python3",
> > "pip3", etc., which is super handy for deploying custom scripts that
> begin
> > with `!#/usr/bin/env python`).
> >
> > I also have a marginally cleaner, ever-so-slightly different version of
> my
> > python package here:
> > https://gist.github.com/zachlewis/8e0b6b0bb089f45b9df7ac85904dcfbd
> > (I prefer this package definition over the one I shared earlier, but
> > someone was having trouble with this one, although I never found what
> > trouble they were having, exactly... would be curious to know if others
> > have trouble. I will also note that I usually build against additional
> 'xz'
> > and sometimes 'libexpat' packages, for native support for tar
compression
> > stuff and xml stuff; and I usually don't bother with 'zlib', unless
> there's
> > some reason to believe the system zlib isn't available).
> >
> > There are also a handful of other python-3 package definitions floating
> > around github, if you have a look... IIRC, @predat has a
> platform-agnostic
> > python-3.6 installer in their rez-repository repo; and I just stumbled
> upon
> > https://github.com/davidlatwe/rez-miniconda which I'm really curious
to
> > try
> > out (although personally, I've only ever used CPython interpreters with
> > rez). Allan also has an example python package in his rez-recipes repo,
> but
> > it's a little ancient...
> >
> > In a perfect world, someone might want to look into hacking [pyenv](
> > https://github.com/pyenv/pyenv) to build and deploy pythons for rez...
> >
> >
> >
> > On Mon, Jan 25, 2021 at 3:54 AM Alberto Sierra <
***@***.***
> >
> > wrote:
> >
> > > I'm facing some of a similar issue.
> > > I'm trying to "bind" both interpreters, python2 and python3. I
> originally
> > > installed Rez using the python2 interpreter. When I try to bind the
> > python3
> > > interpreter using Rez-bind python --exe which python3, It creates the
> > > corresponding package, but when I try to call python within that new
> > > environment, I get the following Python error:
> > >
> > > Fatal Python error: initfsencoding: unable to load the file system
> codec
> > > File
> >
>
"/Users/albertosierra/packages/python/3.7.7/platform-osx/arch-x86_64/os-osx-10.16/python/lib/encodings/__init__.py",
> > line 123
> > > raise CodecRegistryError,\
> > > ^
> > > SyntaxError: invalid syntax
> > >
> > > Current thread 0x000000010d0bbe00 (most recent call first):
> > > [1] 13184 abort python
> > >
> > > What am I doing wrong exactly? Should I omit binding both
interpreters
> > > using Rez-bind and create my own python3 package manually?
> > >
> > > Thanks!
> > >
> > > —
> > > You are receiving this because you commented.
> > > Reply to this email directly, view it on GitHub
> > > <
> >
>
#1016 (comment)
> > >,
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AARAC4F6T6ADUCWUQK2QVNTS3UWUFANCNFSM4WPJAWDA
> > >
> > > .
> > >
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <
>
#1016 (comment)
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAMOUSV4DWLGTJPPZLLLRGDS3WVDNANCNFSM4WPJAWDA
> >
> > .
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <
#1016 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AARAC4D5AAH4H2ZMQACXNTLS346G5ANCNFSM4WPJAWDA
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1016 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSUBJPJSISWU5Z7HQK3S4AR6VANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Just spitballing... Maybe this is a little much, but... what if we could define "pseudo provided package definitions" inline? Yeah, probably too much. Very interesting stuff, my head is spinning with possibilities. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah it starts getting freaky. But it might be the right way to go. I've
thought this kind of thing before. Like, what about when we have package
'features' and you want to build against a debug version of python (that
has a 'build_type-debug' feature for eg), _but_ you've also installed a
debug python build into the system, and you want to use that? Just a simple
provides mechanism doesn't allow for that, because there's nowhere to
define the features that the provided package has.
However, I also think this stuff can get really complicated and it's better
to go one step at a time. The above approach would not be incompatible with
the simpler mechanism described previously (it would just add to it) so we
should add the simpler feature first. Also, perhaps we're overthinking it -
if you really need provided packages to be more expressive (eg have
features) then you can always create an actual package to do that. Eg maya
could require 'python-2.7.13-maya'. It's clunky but it would work.
A
…On Fri, Jan 29, 2021 at 2:58 PM zachlewis ***@***.***> wrote:
Just spitballing...
Maybe this is a little much, but... what if we could define "pseudo
provided package definitions" inline?
# in nuke package.py
@provides('python-2.7.13')
def provided_python():
root='{provider.root}/path/to/local/python/root'
foo = 'bar'
if int(str(provider.version.major)) > 11:
version = '2.7.16'
def commands():
alias( '{provider.name}-{name}', '{root}/python')
if building:
env.PYTHON_EXECUTABLE = '{root}/python'
env.CFLAGS.set('-I{root}/include $CFLAGS')
env.LDFLAGS.set('-L{root}/lib -Wl,{root}/lib $LDFLAGS')
Yeah, probably too much.
Very interesting stuff, my head is spinning with possibilities.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1016 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSWGMFK4UAOJ7ZAAVCTS4IW5FANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Heya — I actually haven’t been maintaining that gist; butI did run into
trouble with the python package and recent macOS versions. I can’t remember
how I sorted that out, actually, but whatever I did was definitely a hack.
When I have the opportunity + bandwidth, I’ll take another look at this;
but I don’t have an immediate solution for you. (But please let me know if
you come up with something before I do!)
In the past, I’ve looked at what pyenv and homebrew are doing, if that
helps…
…On Thu, Aug 19, 2021 at 3:16 PM jlgerber ***@***.***> wrote:
Hi Zach,
At this point, neither the python nor zlib package.pys seem to work for me.
I suspect that this has to do with how literals are (or aren't handled at
this point). I got zlib to sort of work by protecting various vars with the
literal func . However, python doesnt build.
Any change that you have been keeping these up to date and just have not
updated the gists?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1016 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARAC4FKN63SUS3V634SEB3T5VKAHANCNFSM4WPJAWDA>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks for all the info on here, been really useful. I still need to get my head around how I'm going to handle the "provides" functionality while waiting on an official solution. In the mean time I've been working to clean up our python package. There are quite a few helpful repos here https://github.com/OSS-Pipeline (I don't think this is maintained but gives some useful examples). I was able to build gcc and cmake from source which was my biggest hurdle. It also presented a fairly easy template for building packages. I've tried to avoid using any yum installs at all costs and making sure the entire stack is in rez. This brings up the question of an "official" package repo. In my opinion the maintenance of the actual packages is way more valuable day to day than extra features in rez itself. The main issues that we have are not normally "rez can't do that" but "how do a build this package". I believe that a curated package library would dramatically decrease the barrier to entry for rez and seriously reduce the overhead of maintaining a rez environment. Even if it just covered vfxplatform that would be a huge leg up. Forgive me if there is something like this, I heard talk of it a while back. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Agree RE official package repo.
I do have this, but it isn't maintained and it's LGPL (boo!):
https://github.com/nerdvegas/rez-recipes
If there is sufficient interest I would be happy to host a new pkg repo on
my github so it sits alongside rez (and would continue to do so if and when
rez moves elsewhere, to a GH organization for eg).
I would suggest:
* We make its license as permissive as possible - probably MIT/BSD.
Whatever lic removes pretty much any limitation, including keeping
copyright notices intact (who wants to bother with that in package repos).
* We be pretty liberal with ownership. I don't want to become a bottleneck
for merging changes into it.
I think concentrating on the vfx ref platform is absolutely the right idea
for this.
Thx
A
…On Wed, Aug 25, 2021 at 1:50 AM Jared Auty ***@***.***> wrote:
Thanks for all the info on here, been really useful. I still need to get
my head around how I'm going to handle the "provides" functionality while
waiting on an official solution.
In the mean time I've been working to clean up our python package.
Previously our python rez package has been pointing to our system python
but we've decided to bite the bullet and start building python from source
and including it in the rez package.
There are quite a few helpful repos here https://github.com/OSS-Pipeline
(I don't think this is maintained but gives some useful examples). I was
able to build gcc and cmake from source which was my biggest hurdle. It
also presented a fairly easy template for building packages.
The default rez-python package on there wasn't building with things like
sqlite, readline etc so I've been spending the last few days getting all of
python's dependencies together so we have everything we need.
I've tried to avoid using any yum installs at all costs and making sure
the entire stack is in rez.
This brings up the question of an "official" package repo. In my opinion
the maintenance of the actual packages is way more valuable day to day than
extra features in rez itself. The main issues that we have are not normally
"rez can't do that" but "how do a build this package". I believe that a
curated package library would dramatically decrease the barrier to entry
for rez and seriously reduce the overhead of maintaining a rez environment.
Even if it just covered vfxplatform that would be a huge leg up. Forgive me
if there is something like this, I heard talk of it a while back.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1016 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSTTZXPWQUKH6FIRWFTT6O5VTANCNFSM4WPJAWDA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi all, apologies if this isn't the right place to ask questions - please tell me where to otherwise!
What's the correct way to create a python 3 package so I can create a python3 environment and build variants of libraries against both python 2 and 3?
Cheers,
Anders
Beta Was this translation helpful? Give feedback.
All reactions