python and rust interoperability

[johnynek]

The PyO3 library provides a tool, maturin to help in connecting python and rust. Using that tool in a bazel repo opens a lot of new challenged in that it is suggested to be installed with pip and ideally we want a hermetic and reproducible build. Even if that part is tractable, it is still not a solved problem how to link the libraries.

It would be great to have some rules like:

rust_library(
    name = "string_sum",
    srcs = ["string_sum.rs"],
) 

py_rust_library(
    name = "py_string_sum",
    srcs = [], # optionally some additional python code
    deps = [":string_sum"], # either rust_library or py_library dependencies here
)

py_library(
    name = "some_python",
    srcs = ...
)

rust_py_library(
    name = "call_some_py",
    srcs = [], # optionally some rust code here to wrap calling python
    deps = [":some_python", ... ] # we can depend on py_library or rust_library here
)

It seems like putting a solution to this in these rules is the right place for it since I think it would need to interop pretty tightly with rules_rust and maybe less tightly with rules_python (which are still just a wrapper of native python support). On the python side, it seems PyInfo provider may be enough to make things work.

[illicitonion]

@mfarrugi started this out in #753 but it looks like it got a bit stalled. If someone's happy to pick it up and update it, we'll review it :)

[liningpan]

The hardest part for python rust interop is passing correct flags when compiling pyo3, which is typically handled by a build script. The build script calls a python interpreter to gather correct flags. I believe some work needs to be done in rules_python bazel-contrib/rules_python#824.

I have an example here https://github.com/liningpan/pyo3-bazel-example, which allows the pyo3 build script to call a hermetic python interpreter as a work around.

[mfarrugi]

I stopped working on this because there was no way to address the One Definition Rule issue. The the example in: #753 (comment) shows two extensions that each use the helper library, which is then ultimately loaded twice by a python interpreter. It's very possible this is not the true issue underlying the example, and it might just have been an under-baked pull request by me.

The naive implementation is probably workable 90%+ of the time, but otherwise requires careful usage or fully dynamic linking to be safe (including rust std?). For what it's worth, rules_python didn't address the ODR issue either, and I would be surprised if it does now; there are docs somewhere on how this is solved inside google/blaze.

[rickeylev]

(I'm a rules_python maintainer; I was pointed here by someone)

What information from the Python runtime/toolchain is needed? rules_python now has @rules_python//python/cc:current_py_* targets to give the headers and runtime libraries. Does Rust need something else or more?

I stopped working on this because there was no way to address the One Definition Rule issue. The the example in: #753 (comment) shows two extensions that each use the helper library, which is then ultimately loaded twice by a python interpreter. It's very possible this is not the true issue underlying the example, and it might just have been an under-baked pull request by me.

I'd agree with you're assessment, you're probably right.

There's a couple solutions:

1. Have something like cc_shared_library. Basically something that allows creating a shared library while ensuring only the desired pieces are linked into it. Consumers are responsible for making sure any other necessary dependencies are provided. i.e. dependency CcInfo isn't forwarded on.
2. Make py_rust_library create a shared library for its own sources, but forward on any CcInfo from dependencies. Consumers (usually the final binary rule, e.g. py_binary) are responsible for transforming all that CcInfo dependency information into something usable.

For (2), there's some unactivated code in py_binary that takes that dependency information and creates a shared library (grep for "native_deps" in the python code base) that Python should be able to load.

If embedded Python is being used, then statically linking everything is also an option.

HTH

[UebelAndre]

There is now https://github.com/abrisco/rules_pyo3 in case it's helpful to anyone 😄