How to configure PyO3 to respect my venv ( ModuleNotFoundError ) when calling Python from Rust

[ZizoAdam]

Python 3.11.7
Windows 10 and Ubuntu 22.04 via Docker/WSL2

I have tried using pyenv (both on Windows and Ubuntu) and currently Windows which is ultimately where I'd like things to be running but an Ubuntu dev container was convenient at the time as I could just globally install for the time being until I resolved the issue. I have also tried this with Poetry but had the same issue and I couldn't find any way to sort this via manually setting PYO3_PYTHON or using PYO3_CONFIG_FILE

I repeatedly ran into the issue of getting ModuleNotFoundError when importing third party dependencies in my venv so printing some debug information to the console as so

            let sysconfig = PyModule::import(py, "sysconfig").unwrap();
            let python_version = sysconfig.call_method0("get_python_version").unwrap();
            log::debug!("Using python version: {}", python_version);
            let python_lib = sysconfig
                .call_method("get_config_var", ("LIBDEST",), None)
                .unwrap();
            log::debug!("Using python lib: {}", python_lib);
            let python_site_packages = sysconfig
                .call_method("get_path", ("purelib",), None)
                .unwrap();
            log::debug!("Using python site-packages: {}", python_site_packages);

I get

2024-01-04T12:37:42.187Z DEBUG [demo] Using python version: 3.11
2024-01-04T12:37:42.188Z DEBUG [demo] Using python lib: C:\Users\Adam\AppData\Local\Programs\Python\Python311\Lib
2024-01-04T12:37:42.188Z DEBUG [demo] Using python site-packages: C:\Users\Adam\AppData\Local\Programs\Python\Python311\Lib\site-packages

It is at least partially respecting my venv as it's using Python 3.11 rather than 3.12 when I try to run via poetry run but notably seems to be pointing to the wrong site-packages.

How can I point PyO3 to the right place?

[ZizoAdam]

A bodgy workaround that is doing the job at the moment is manually adding the site packages via

import sys
sys.path.append('my\\venv\\site-packages') # obviously not my real filepath

[isonao]

Thank you. When I was about to give up, I found this post and tried it, and it worked. (Windows 10)

Python::with_gil(|py| {
    let sys = py.import_bound("sys")?;
    let path = sys.getattr("path")?;
    path.call_method1("append", ("myvenv\\Lib\\site-packages",))?;  // append my venv path
    
    let requests = py.import_bound("requests")?;    //not error occurred!!
    ......

[davidhewitt]

This detail may help point you in the right direction: #3717 (comment)

[ZizoAdam]

Activating the virtual environment only seems to point it towards the correct python executable (at least on Windows) and it never picks up the correct path for the third party dependencies. I think the more relevant environment variable would be PYTHONPATH but my current bodge job seems to work for now