package.py doesn't get __file__ set #1338
-
|
Hello, I'm trying to do some hackery in a package.py (specifically in a rez-search context) that relies on knowing the path to the package.py, only |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 5 replies
-
|
Line 223 in 682edda I have to tell you to be careful if you really want to add a |
Beta Was this translation helpful? Give feedback.
-
|
It's actually exploring the relative path (i.e. the variants) underneath the package repo, so as long as relative paths stay the same, all will be well. Thanks for the pointers! |
Beta Was this translation helpful? Give feedback.
-
|
Even if you check relative paths, variant paths can be hashed, which makes them useless... |
Beta Was this translation helpful? Give feedback.
-
|
All that to say, it is unsafe for a package to know about a package repository's structure. |
Beta Was this translation helpful? Give feedback.
-
|
I'll bear that in mind, thanks for the advice! |
Beta Was this translation helpful? Give feedback.
-
|
Can you share some more information about what you are trying to do? Exploring variants and variant folders seems something you may want to do programatically using the rez API rather than vanilla python. |
Beta Was this translation helpful? Give feedback.
-
|
Echoing others' points here. Any assumptions you make about variants being
somewhere on the filesystem under the package 'base' dir will tie you to
that package repo impl, which is bad.
Specifically wrt `__file__`, JC is right and in fact nothing stops a studio
currently having an (eg) database pkg repo impl, where `__file__` is
meaningless, and the package.py never exists on disk.
In general you should:
* assume nothing about the location of a package.py;
* don't assume anything about variants' relative locations to one another,
nor to the package 'base'. You can only know what the 'root' of a given
variant is, nothing more.
* don't assume anything about a known variant 'root' path. For eg, you
cannot assume that pkg requirements make up the path structure
Please note that the docs here
https://github.com/AcademySoftwareFoundation/rez/wiki/Variants#disk-structure,
while correct, are specific to the "filesystem" pkg repo type. We need to
update this to make that obvious and we should probably state there the
kind of stuff we're talking about here in this thread (ie don't assume
structure etc etc).
Hth
A
…On Mon, Jun 27, 2022 at 4:50 AM Thorsten Kaufmann ***@***.***> wrote:
Can you share some more information about what you are trying to do?
Exploring variants and variant folders seems something you may want to do
programatically using the rez API rather than vanilla python.
—
Reply to this email directly, view it on GitHub
<#1338 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSQMGRCDN74WP6WUAVDVRCQ7PANCNFSM5Z22WRZA>
.
You are receiving this because you are subscribed to this thread.Message
ID: <AcademySoftwareFoundation/rez/repo-discussions/1338/comments/3027119@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks everyone! In this specific case I actually *can* assume the
structure of the package repo since it's not really a package repo but a
repo of recipes which I can guarantee to have a certain structure. I'm
using rez-search to scan the recipes repo and I'd like the package.py's it
finds to enumerate the various variants that recipes exist for under that
structure. It's a bit of a hack (and it's very slow traversing a dependency
tree with many calls to rez-search) but it basically works for what I need
for now. Longer term I'll want to find a better solution.
Cheers,
Anders
…On Mon, 27 Jun 2022 at 12:28, allan johns ***@***.***> wrote:
Echoing others' points here. Any assumptions you make about variants being
somewhere on the filesystem under the package 'base' dir will tie you to
that package repo impl, which is bad.
Specifically wrt `__file__`, JC is right and in fact nothing stops a studio
currently having an (eg) database pkg repo impl, where `__file__` is
meaningless, and the package.py never exists on disk.
In general you should:
* assume nothing about the location of a package.py;
* don't assume anything about variants' relative locations to one another,
nor to the package 'base'. You can only know what the 'root' of a given
variant is, nothing more.
* don't assume anything about a known variant 'root' path. For eg, you
cannot assume that pkg requirements make up the path structure
Please note that the docs here
https://github.com/AcademySoftwareFoundation/rez/wiki/Variants#disk-structure
,
while correct, are specific to the "filesystem" pkg repo type. We need to
update this to make that obvious and we should probably state there the
kind of stuff we're talking about here in this thread (ie don't assume
structure etc etc).
Hth
A
On Mon, Jun 27, 2022 at 4:50 AM Thorsten Kaufmann ***@***.***>
wrote:
> Can you share some more information about what you are trying to do?
> Exploring variants and variant folders seems something you may want to do
> programatically using the rez API rather than vanilla python.
>
> —
> Reply to this email directly, view it on GitHub
> <
#1338 (reply in thread)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAMOUSQMGRCDN74WP6WUAVDVRCQ7PANCNFSM5Z22WRZA
>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID:
<AcademySoftwareFoundation/rez/repo-discussions/1338/comments/3027119@
> github.com>
>
—
Reply to this email directly, view it on GitHub
<#1338 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOYQXICPBQXNRN7VVLYVLDVRDYRFANCNFSM5Z22WRZA>
.
You are receiving this because you authored the thread.Message ID:
<AcademySoftwareFoundation/rez/repo-discussions/1338/comments/3027864@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Specifically if you look here you can see the recipes repo: https://github.com/anderslanglands/rez-recipes If you look under e.g. openexr-3.1.5 you can see that it has variants for linux and windows. I'd like the |
Beta Was this translation helpful? Give feedback.
-
|
Something to think about is the approach to recipes taken in
#673.
My thinking on how this will work is that a package.py (unbuilt that is)
will construct its definition in response to the build request. So for
example, a package will present with the variant that should be built, like
so:
(note: pseudocode)
```
@early()
def variants():
return [["platform-" + system.platform, "os-" + system.os]]
```
So the package.py will always have just one variant, as far as the build is
concerned. The installed package will have any number though, since variant
installs are additive.
Note that one thing we need to do is add a new rez-build arg, I'm thinking
of calling it "constraints" perhaps, not sure. Consider a case where a
package can be built with any version of say, openexr-3.*, but once built,
it has a specific dependency on that version. We have an approach to doing
this (that's still in a PR atm). Impl aside though, you need a way to
specify the openexr version you want instead of just getting the latest of
openexr-3.*. So you need do be able to do something like so:
]$ rez-build --constraints openexr-3.1.5
Now that we have ephemerals (and features, which are still to come), you
can think of 'constraints' as being more than just extra requirements for
the build runtime. Rather, they will also be general configuration used to
drive the build, with attributes such as build_type and so on. I could see
that getting verbose enough that we'd want to add support for a new kind of
"build configuration" config file for just this purpose, _but_ the end
result would be the same (ie the config file could also be expressed with
equivalent constraint args instead, if one desired).
A
…On Mon, Jun 27, 2022 at 12:22 PM Anders Langlands ***@***.***> wrote:
Specifically if you look here you can see the recipes repo:
https://github.com/anderslanglands/rez-recipes
If you look under e.g. openexr-3.1.5 you can see that it has variants for
linux and windows. I'd like the openexr/3.1.5/package.py to scan the
variant tree underneath it to build its list of variants, otherwise I have
to be careful to keep that directory structure and
openexr/3.1.5/package.py in sync.
—
Reply to this email directly, view it on GitHub
<#1338 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSQ7N5A3NS25FULQL73VREF5NANCNFSM5Z22WRZA>
.
You are receiving this because you commented.Message ID:
<AcademySoftwareFoundation/rez/repo-discussions/1338/comments/3028216@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Yep I've implemented constraints already. The trouble with having just a single package.py that builds all variants is that the build process (and the final package.py's) can be so different for different platforms that trying to combine them in a big "if" is a horrible mess. It's much cleaner to separate them. |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps what we really need is a natively supported way to have a
package.py be able to forward onto another defn file during the build?
Something like so:
```
# in package.py
# these get inherited by the other defn files (eg windows.py)
name = "openexr"
description = "..."
if system.platform == "windows"
load_definition("windows.py")
else:
load_definition("not_windows.py")
```
…On Mon, Jun 27, 2022 at 1:47 PM Anders Langlands ***@***.***> wrote:
Yep I've implemented constraints already.
The trouble with having just a single package.py that builds all variants
is that the build process (and the final package.py's) can be so different
for different platforms that trying to combine them in a big "if" is a
horrible mess. It's much cleaner to separate them.
—
Reply to this email directly, view it on GitHub
<#1338 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMOUSU46HVAXGKDQVEMG4TVREP6NANCNFSM5Z22WRZA>
.
You are receiving this because you commented.Message ID:
<AcademySoftwareFoundation/rez/repo-discussions/1338/comments/3028456@
github.com>
|
Beta Was this translation helpful? Give feedback.
__file__isn't set because the package definitions are not imported using the Python import system. They are opened, read, compiled and then executed and the result is stored in a variable that contains all variables defined in the package definition.rez/src/rez/serialise.py
Line 223 in 682edda
I have to tell you to be careful if you really want to add a
__file__variable. I don't think we can rely on the path of the definition file (package.py). For example a package can be moved from one repository to another (for archival purposes, multi-site transfers, etc), it could be locally cached or hell in the future could even live on somet…