Issue with reticulate::py_module_available("rdkit") returning TRUE in RStudio Console but FALSE in Shiny Server.R

[bshashikadze]

I'm encountering strange behavior using the reticulate package in R. When I run the following in the RStudio console:

reticulate::py_module_available("rdkit")

It returns TRUE, but when I include the same code at the top of my server.R in a Shiny app, run the app, it prints FALSE.

However, if I first run reticulate::py_module_available("rdkit") in the R console and then start the Shiny app, the code inside Shiny prints TRUE as expected. But if I start a fresh Shiny session, it prints FALSE.

Interestingly, for other modules like matplotlib, I don’t see this issue: both in the console and inside the Shiny app, reticulate::py_module_available("matplotlib") returns TRUE.

I also verified that both libraries (rdkit and matplotlib) are located in the same directory.

Has anyone encountered this issue or know why rdkit behaves differently in a Shiny app compared to the R console?

Many thanks in advance

Bachi

[t-kalinowski]

What does print(py_config()) output in the two contexts?

Most likely, a different Python installation is being used, either due to a different working directory, or different environment variables.

Consulting the Order of Discovery might be helpful

[bshashikadze]

Thank you very much for responding

RStudio console:
reticulate::py_config()
python: /usr/local/bin/python3.12
libpython: /usr/local/lib/libpython3.12.so
pythonhome: /usr/local:/usr/local
version: 3.12.0 (main, Sep 2 2024, 15:16:03) [GCC 11.4.0]
numpy: /usr/local/lib/python3.12/site-packages/numpy
numpy_version: 2.0.2

NOTE: Python version was forced by use_python() function

reticulate::py_module_available("rdkit")
[1] TRUE

Shiny server.R
reticulate::py_config()
python: /usr/local/bin/python3.12
libpython: /usr/local/lib/libpython3.12.so
pythonhome: /usr/local:/usr/local
version: 3.12.0 (main, Sep 2 2024, 15:16:03) [GCC 11.4.0]
numpy: /usr/local/lib/python3.12/site-packages/numpy
numpy_version: 2.0.2

NOTE: Python version was forced by use_python() function

reticulate::py_module_available("rdkit")
[1] FALSE

Interestingly if I take two step approach and run reticulate::py_module_available("rdkit") in console first and then open shiny app, reticulate::py_module_available("rdkit") (from shiny server.R) prints TRUE

[t-kalinowski]

Where is rdkit being loaded from in the successful case?

module <- import("rdkit")
py_repr(module)
py_get_attr(module, "__path__")
py_get_attr(module, "__file__")

[bshashikadze]

py_repr(module)
[1] "<module 'rdkit' from '/usr/local/lib/python3.12/site-packages/rdkit/init.py'>"
py_get_attr(module, "__path__")
['/usr/local/lib/python3.12/site-packages/rdkit']
py_get_attr(module, "__file__")
'/usr/local/lib/python3.12/site-packages/rdkit/init.py'

[t-kalinowski]

Interestingly if I take two step approach and run reticulate::py_module_available("rdkit") in console first and then open shiny app, reticulate::py_module_available("rdkit") (from shiny server.R) prints TRUE

I think this is a hint that the seemingly identical py_config() outputs are incorrect. Most likely, the py_config() call itself is forcing Python to load, before the shiny app code has run and modified python discovery.

Try this. In the shiny app, at the first location where python is initialized by the shiny app:

tryCatch(import("rdkit"), error = function(e) {
  print(py_config())
  stop(e)
})

[bshashikadze]

Thank you very much for your patience ;)

thats what I am getting:

python: /usr/local/bin/python3.12
libpython: /usr/local/lib/libpython3.12.so
pythonhome: /usr/local:/usr/local
version: 3.12.0 (main, Sep 2 2024, 15:16:03) [GCC 11.4.0]
numpy: /usr/local/lib/python3.12/site-packages/numpy
numpy_version: 2.0.2
rdkit: /usr/local/lib/python3.12/site-packages/rdkit

NOTE: Python version was forced by use_python() function
Error in py_module_import(module, convert = convert) :
ModuleNotFoundError: No module named 'rdkit.Chem'; 'rdkit' is not a package
Run reticulate::py_last_error() for details.

[t-kalinowski]

Can you provide an MRE?

It could be a binary incompatibility between rdkit and something in the shiny app, where loading the app causes R to load a dll that then prevents rdkit from loading correctly. We've observed that before between data.table loading libomp and PyTorch throwing an error afterwards. The workaround there (short of rebuilding R or PyTorch to use the same libomp), is to simply load Pytorch before loading data.table.

Is pre-loading Python + rdkit before loading shiny an option?

[bshashikadze]

While creating the MRE, I noticed that print(reticulate::py_module_available("rdkit")) returned TRUE on the server, regardless of whether it was run before or after loading Shiny or other libraries.

The app (actual one which was failing) includes a .py file, though it's not explicitly sourced using reticulate::source_python(). Coincidentally, I had named this file rdkit.py, which seemingly conflicted with the library name. Renaming the file resolved the issue. I also observed that the pycache folder was created when RDKit wasn't found, but after renaming the file, it no longer appeared.

[t-kalinowski]

Seems like you were masking the installed rdkit python package with a rdkit.py file in the current working directory.

Thanks for reporting back and closing the loop!