feat: change PYBIND11_EMBEDDED_MODULE to multiphase init

[b-pass]

Description

Changes PYBIND11_EMBEDDED_MODULE to do multiphase init. This is required for embedded modules to properly express their support (or not) for free threading and multiple interpreters. Also added a note to the docs for this.

I believe this will also fix the free-threading multiple interpreter test failed @henryiii had on #5646.

Suggested changelog entry:

* Changed ``PYBIND11_EMBEDDED_MODULE`` macro implementation to perform multi-phase module initialization (PEP 489) behind the scenes and to support `py::mod_gil_not_used()`, `py::multiple_interpreters::per_interpreter_gil()` and `py::multiple_interpreters::shared_gil()`.

[Copilot]

Pull Request Overview

This PR updates the PYBIND11_EMBEDDED_MODULE macro to perform multiphase module initialization, enabling support for free threading and multiple interpreters, and updates associated documentation and CI testing.

• Update of the PYBIND11_EMBEDDED_MODULE macro to accept multiphase init flags
• Test adjustments to validate per-interpreter GIL behavior
• Documentation updates to describe the new initialization options and CI configuration changes

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/test_embed/test_interpreter.cpp	Adjusts tests to check behavior of modules with and without per_interpreter_gil
include/pybind11/embed.h	Updates macro implementation to support multiphase init and new return types
docs/advanced/misc.rst	Adds notes and anchor references for free-threading and sub-interpreter support
docs/advanced/embedding.rst	Updates documentation to list accepted multiphase init flags
.github/workflows/ci.yml	Updates CI matrix to include new Python versions with 't' suffixes

Comments suppressed due to low confidence (2)

tests/test_embed/test_interpreter.cpp:519

• [nitpick] The change from 'widget_module' to 'trampoline_module' in the failing import test may be confusing. Please ensure that 'trampoline_module' is properly defined and its naming clearly distinguishes it from 'widget_module' in this context.

py::module_::import("trampoline_module");

include/pybind11/embed.h:70

• The multiphase module initialization now returns an int instead of a module pointer. Please confirm that all consumers of this macro handle the updated return type consistent with PEP 489 conventions.

return 0;

[henryiii]

It is passing on 3.13t and 3.14t!

[ericsnowcurrently]

I'm not super familiar with pybind11 or C++, but this makes sense to me. You've set up the module def with the slots and added the exec func, which is correct.

[henryiii]

The open jobs are taking 3+ hours (timeout is the default 6 hours). Something here is causing it to hang when it gets to the embedding test. I think all the hanging ones are <3.12. That's also what happened to Appveyor.

[rwgk]

Drive by comment. (I need to look more closely later.)

[rwgk]

The open jobs are taking 3+ hours (timeout is the default 6 hours). Something here is causing it to hang when it gets to the embedding test. I think all the hanging ones are <3.12. That's also what happened to Appveyor.

I was able to reproduce locally using miniforge3, roughly:

conda create --yes -n py310 python=3.10
conda activate py310

cmake -S $HOME/forked/pybind11 -B cmake_build_conda_py310 -DCMAKE_VERBOSE_MAKEFILE=ON -DDOWNLOAD_CATCH=ON -DPYTHON_EXECUTABLE=$(which python3) -DCMAKE_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_PYTEST_ARGS=-vv && cd cmake_build_conda_py310 && make check -j 64

This hangs. Then for debugging I switched to my scons-based build, which I can more easily navigate. It also hangs. I ran some observations by ChatGPT, look from the bottom up here (i.e. ignore the random stuff at the top):

https://chatgpt.com/share/6828e3a2-bcac-8008-889a-88f68c3eeba4

[henryiii]

I thought that might be the problem, it was very suspicious that it was passing exactly on the version that support subinterpreters.

[b-pass]

Not sure what to do about the embedded "threading" test. I thought the critical section would fix it, but it apparently didn't. It's more of a GIL test, we could just ifdef it out entirely.... (The failure is not related to sub-interpreters or multiphase init, it's a normal free-threading problem, there's no lock on the dict it uses....)

[henryiii]

Do we know why it works on 3.14t but not 3.13t?

[henryiii]

@colesbury Do you know why Py_CRITICAL_SECTION(dict) works in 3.14t but not in 3.13t? Is that expected?

I worked around it by using a std::mutex for 3.13t.

Tested locally with:

rm -rf build .venv
cmake --preset venv -DPYBIND11_CREATE_WITH_UV=3.13t
cmake --build --preset venv --target cpptest

rm -rf build .venv
cmake --preset venv -DPYBIND11_CREATE_WITH_UV:PATH=~/.pyenv/versions/3.14.0b1t/bin/python
cmake --build --preset venv --target cpptest

(#5668 swaps the rm for support for the built-in cmake --fresh)