docs of deprecated C function should propose replacement in C, not in Python

[jwuttke]

Documentation

In the 3.13 Python/C API Reference Manual, page "Initialization, Finalization, and Threads", Py_GetProgramName() and several other C functions are declared deprecated. To replace them, it is suggested "Get sys.executable instead".

As maintainer of a C program, I find this not helpful. I would need a drop-in replacement in C, not a reference to a Python variable.

Linked PRs

• gh-129342: Explain how to replace Py_GetProgramName() in C #129361

[serhiy-storchaka]

Like PySys_GetObject("executable")?

[jwuttke]

    PyObject* obj = PySys_GetObject("executable");
    PyObject* repr = PyObject_Repr(obj);
    PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
    const std::string s = PyBytes_AS_STRING(str);

kind of works, but it returns the full path, not just the executable name.

[serhiy-storchaka]

Don't use PyObject_Repr(), it does not do what you think it does.

PyUnicode_AsUTF8AndSize() can be more convenient than PyUnicode_AsEncodedString() + PyBytes_AS_STRING() to you.

I suggested PySys_GetObject("executable") as a way to get the sys.executable value. If it is not equivalent to the Py_GetProgramName() result, the suggestion in the documentation is not correct. cc @vstinner

[jwuttke]

Thanks.

These replacements for Py_GetProgramFullPath and Py_GetPath work for me:

    {
	PyObject* obj = PySys_GetObject("executable");
	PyObject* repr = PyObject_Str(obj);
	const std::string s = PyUnicode_AsUTF8AndSize(repr, NULL);
	std::cout << "  - executable: " + s + "\n";
    }
    {
	PyObject* obj = PySys_GetObject("path");
	PyObject* repr = PyObject_Str(obj);
	const std::string s = PyUnicode_AsUTF8AndSize(repr, NULL);
	std::cout <<  "  - Python PATH: " + s + "\n";
    }

Replacement of Py_GetPythonHome seems to require a somewhat different solution.

[jwuttke]

https://docs.python.org/3/c-api/init.html#c.Py_GetPythonHome says: "Get PyConfig.home or PYTHONHOME environment variable instead."

Retrieving a Python enviroment variable from C seems feasible but lengthy. So I tried PyConfig.home:

    {
	PyConfig config;
	PyConfig_InitPythonConfig(&config);
	auto status = PyConfig_Read(&config);
	if (PyStatus_Exception(status))
	    throw std::runtime_error("Cannot access PyConfig");
	std::cout << "  - Python home: " + toString(config.home) + "\n";
	PyConfig_Clear(&config);
    }

Seems to work. Is there a simpler solution?

[jwuttke]

@serhiy-storchaka I just see on PEP731 that you are member of the C API working group. So you are the right person to ask: WHY these deprecations? They break production code, and now it turns out that there is not even a 1:1 replacement...

[vstinner]

Retrieving a Python enviroment variable from C seems feasible but lengthy. So I tried PyConfig.home: (...)

I added PyConfig_Get() from PEP 741 "Python Configuration C API" to Python 3.14. The pythoncapi-compat project makes the function available on Python 3.13 and older. Would it be a good replacement for your use case?

Py_GetPythonHome() can be replaced with PyConfig_Get("home").

[jwuttke]

Of course a 1:1 replacement is much better than a multi-line workaround.

Is there a similarly simple solution for Py_GetProgramFullPath and Py_GetPath?

Nonethess, my question goes: WHY these deprecations? Have you really well pondered aginst the damage done (not least to Python's reputation) by breaking things?

[vstinner]

• Py_GetProgramFullPath() can be replaced with PyConfig_Get("executable") or PySys_GetObject("executable").
• Py_GetPath() can be replaced with PyConfig_Get("module_search_paths") or PySys_GetObject("path").

Would you mind to elaborate on your use case? You call Py_GetProgramFullPath(), Py_GetPath() and Py_GetPythonHome() after Python initialization? What is the purpose of reading these variables?

Nonethess, my question goes: WHY these deprecations? Have you really well pondered against the damage done (not least to Python's reputation) by breaking things?

In Python 3.8, I designed and implemented PEP 587 "Python Initialization Configuration" which unifies the "Python initialization". Slowly (since Python 3.8), I deprecated the old way to configure Python.

How can I help you to migrate your code towards newer APIs which are not deprecated? Does PySys_GetObject() and/or PyConfig_Get() answer to your question?

[serhiy-storchaka]

Sorry, it was discussed before I became the member of the C API working group, so I am not aware of arguments. I guess this is related to handling global state at different stages of Python initialization and shutdown, and maybe to supporting multiple interpreters.

But we should be more conservative with deprecating the public C API. For every deprecated function we should document an alternative that works in all maintained Python versions. If PyConfig_Get("home") is the only alternative to Py_GetPythonHome(), the latter should not be deprecated until all maintained Python versions have PyConfig_Get(), i.e. not earlier than 3.17.

[jwuttke]

My use case: in a complex software with C++ core and Python API [1,2], we want to introspect the state of an embedded Python interpreter for various debugging purposes.

Anyway, functions Py_GetProgramName() etc have been part of the public API, and you should not be surprised that they have been used and that production code depends on them.

I had hoped that the Python community had learned from the forced upgrade from Py2 to Py3. But no, you are breaking things within major 3, with relatively short transition periods. This is in stark contrast with the emphasis C++ is putting on backwards compatibility, and it raises serious doubts whether C++ and Python is a good combination in a software project that is meant to be maintained for decades to come.

This said, a well-documented 1:1 drop-in replacement like PyConfig_Get("home") for Py_GetPythonHome() is much much better than my above three-line replacement for Py_GetProgramFullPath.

But why not leave Py_GetPythonHome() in the API for the lifetime of Python major 3? You may still reimplement it internally as a trivial wrapper of PyConfig_Get("home").

[1] https://www.bornagainproject.org/
[2] https://jugit.fz-juelich.de/mlz/bornagain