Undefined references to pyo3 structures when building a shared library on Linux?

[golmschenk]

Hello! I'm attempting to build a library that will allow Fortran to call Python through Rust using pyo3. During the build, I encounter a undefined reference to 'PyExc_BaseException' along with many other undefined reference exceptions. I have built a minimal example of this situation. Due to the Fortran compiling, the project uses CMake and Corrosion to build the Rust pyo3-based library. The build works on macOS, but fails on Linux.

The docs troubleshooting notes a similar error, but this is for extension modules explicitly. Due to a warning: libpython3.11.so.1.0, needed by /...../librust_library.so, not found (try using -rpath or -rpath-link) message, it seems likely this might be an issue with the paths for the lookup. I had originally included the setting of the rpath to get the build to work on macOS, which worked successfully. Since then, I've tried various ways to set the rpath or other various target linking methods in the CMakeLists.txt to help libpython be discovered, however, I've been unsuccessful. I separately built a minimal case of just Rust calling Python still using CMake and Corrosion to perform the build. This works successfully on both macOS and Linux. Though there's a good chance I'm just not setting the pathing correctly during the build, due to the somewhat similar known issue in the troubleshooting docs, I wanted to check here as well that it might not be some other issue.

If anyone can clarify my mistake in setting up the correct rpaths for the library, or can point out why this might be a different issue, I would greatly appreciate it! Thank you for your time!

CMakeLists.txt build script:

cmake_minimum_required(VERSION 3.17)

project(fortran_calling_python_through_rust_using_pyo3 LANGUAGES Fortran)

find_package(Python REQUIRED Interpreter Development)

include(FetchContent)

FetchContent_Declare(
        Corrosion
        GIT_REPOSITORY https://github.com/corrosion-rs/corrosion.git
        GIT_TAG v0.4
)

FetchContent_MakeAvailable(Corrosion)

corrosion_import_crate(MANIFEST_PATH Cargo.toml)
corrosion_add_target_local_rustflags(rust_library "-Clink-args=-Wl,-rpath,${Python_LIBRARY_DIRS}")

add_executable(fortran_executable src/main.f90)
target_link_libraries(fortran_executable rust_library)

Build output:

% cmake --build build --config Debug
[  0%] Built target cargo-prebuild_rust_library
   Compiling target-lexicon v0.12.13
   Compiling autocfg v1.1.0
   Compiling proc-macro2 v1.0.76
   Compiling once_cell v1.19.0
   Compiling libc v0.2.152
   Compiling unicode-ident v1.0.12
   Compiling parking_lot_core v0.9.9
   Compiling cfg-if v1.0.0
   Compiling scopeguard v1.2.0
   Compiling smallvec v1.11.2
   Compiling heck v0.4.1
   Compiling indoc v2.0.4
   Compiling unindent v0.2.3
   Compiling lock_api v0.4.11
   Compiling memoffset v0.9.0
   Compiling quote v1.0.35
   Compiling pyo3-build-config v0.20.2
   Compiling syn v2.0.48
   Compiling parking_lot v0.12.1
   Compiling pyo3-ffi v0.20.2
   Compiling pyo3 v0.20.2
   Compiling pyo3-macros-backend v0.20.2
   Compiling pyo3-macros v0.20.2
   Compiling rust_library v0.1.0 (/local/data/emussd1/golmsche/language_interoperability/fortran_calling_python_through_rust_using_pyo3)
    Finished dev [unoptimized + debuginfo] target(s) in 8.90s
Copying byproducts `librust_library.so` to /local/data/emussd1/golmsche/language_interoperability/fortran_calling_python_through_rust_using_pyo3/build
[  0%] Built target _cargo-build_rust_library
[  0%] Built target cargo-build_rust_library
[ 50%] Building Fortran object CMakeFiles/fortran_executable.dir/src/main.f90.o
[100%] Linking Fortran executable fortran_executable
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libpython3.11.so.1.0, needed by /local/data/em
ussd1/golmsche/language_interoperability/fortran_calling_python_through_rust_using_pyo3/build/librust_library.so, not found (try using -rpath or -rpath-link)
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyExc_BaseException'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyImport_Import'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyUnicode_AsEncodedString'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyGILState_Release'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyBytes_Size'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyExc_TypeError'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `Py_InitializeEx'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyObject_Str'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyUnicode_AsUTF8AndSize'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyFloat_FromDouble'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyObject_GetAttr'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyFloat_Type'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_SetObject'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyBytes_AsString'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyEval_SaveThread'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_Restore'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyType_IsSubtype'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_NewExceptionWithDoc'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyGILState_Ensure'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_NormalizeException'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_Fetch'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyFloat_AsDouble'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyObject_Call'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_Print'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `_Py_Dealloc'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyTuple_New'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_SetString'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyUnicode_FromStringAndSize'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyErr_PrintEx'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `Py_IsInitialized'
/local/data/emussd1/golmsche/miniforge3/envs/eesunhong_dev_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /local/data/emussd1/golmsche/language_interoperability/
fortran_calling_python_through_rust_using_pyo3/build/librust_library.so: undefined reference to `PyObject_Repr'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/fortran_executable.dir/build.make:97: fortran_executable] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:263: CMakeFiles/fortran_executable.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

[davidhewitt]

Sorry for the long delay here. If you're still stuck on this, it might be worth trying to run the compilation with PYO3_PRINT_CONFIG=1 environment variable set. This might help debug what configuration PyO3 is picking up during the build.

[golmschenk]

Finally got around to testing this, but I can no longer reproduce my original issue. Since my original question, I had cleared my Rust installation and reinstalled it (via rustup). I don't know if this or something else resolved the issue. But I no longer seem to be encountering the issue regardless. Thank you for your time!