[feat/precompile-tooling] docs: document the opt-in precompiled mode

.pre-commit-config.yaml

@@ -141,6 +141,11 @@ repos:
     language: pygrep
     entry: PyBind|\bNumpy\b|Cmake|CCache|PyTest
     exclude: ^\.pre-commit-config.yaml$
+  - id: check-inl-headers
+    name: Check -inl.h files for the precompiled mode
+    language: python
+    entry: python tools/check_inl_headers.py
+    files: -inl\.h$
 
 # PyLint has native support - not always usable, but works for us
 - repo: https://github.com/PyCQA/pylint

docs/compiling.rst

@@ -348,7 +348,8 @@ function with the following signature:
 .. code-block:: cmake
 
     pybind11_add_module(<name> [MODULE | SHARED] [EXCLUDE_FROM_ALL]
-                        [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] source1 [source2 ...])
+                        [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] [PRECOMPILE | NO_PRECOMPILE]
+                        source1 [source2 ...])
 
 This function behaves very much like CMake's builtin ``add_library`` (in fact,
 it's a wrapper function around that command). It will add a library target
@@ -404,6 +405,104 @@ optimizations remain disabled.
 
 .. _ThinLTO: http://clang.llvm.org/docs/ThinLTO.html
 
+.. _precompile-mode:
+
+Pre-compiling part of pybind11
+------------------------------
+
+pybind11 is header-only by default: every translation unit compiles its own
+copy of the non-template implementation. The opt-in *precompiled* mode
+compiles that implementation once, into a static library built inside your
+own project with your own flags. This reduces the build time, most of all for
+projects with many translation units or many modules in one build.
+
+.. code-block:: cmake
+
+    pybind11_add_module(example PRECOMPILE example.cpp)
+
+The first ``PRECOMPILE`` target creates the library target
+``pybind11::precompiled``; further targets reuse it. Set the CMake variable
+``PYBIND11_PRECOMPILE`` to make it the default for all
+``pybind11_add_module`` calls; use ``NO_PRECOMPILE`` on a target to opt back
+out. For targets you create yourself, call the ``pybind11_precompile()``
+function and link ``pybind11::precompiled`` PRIVATE; the target carries the
+required ``PYBIND11_PRECOMPILED`` compile definition PUBLIC, so your sources
+also get it.
+
+Requirements and caveats:
+
+* The library and every module linking it must agree on the configuration
+  macros ``PYBIND11_INTERNALS_VERSION``, ``Py_GIL_DISABLED``,
+  ``PYBIND11_SIMPLE_GIL_MANAGEMENT``,
+  ``PYBIND11_DETAILED_ERROR_MESSAGES`` (defaults on in debug builds),
+  ``PYBIND11_HAS_SUBINTERPRETER_SUPPORT``, and
+  ``PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET``. A mismatch produces one
+  readable undefined symbol at link time referencing
+  ``pybind11_precompiled_config``.
+* Configuration macros that only change code inside the library (for example
+  ``PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING``) must be defined when the
+  library is compiled; a definition only on your module has no effect. This
+  also applies to a custom ``PYBIND11_NAMESPACE`` visibility attribute: set
+  it at the directory level (``add_compile_definitions``) so that the library
+  gets it too.
+* The library picks up your directory-level flags and C++ standard when it is
+  first created, so set those before the first ``PRECOMPILE`` target. A
+  status message reports the directory that created the library.
+* The library is not compiled with link-time optimization, and the per-target
+  ``THIN_LTO`` and ``OPT_SIZE`` options of ``pybind11_add_module`` do not
+  apply to it. In a Release build this adds a few nanoseconds to each call of
+  a bound function (up to about 10% for a function that does nothing,
+  depending on the compiler; a few percent with LTO on the library). To change this, call ``pybind11_precompile()``
+  yourself and
+  set the properties on the created target, ``pybind11_precompiled`` (the
+  real target behind the ``pybind11::precompiled`` alias; CMake does not let
+  you set properties through an alias):
+
+  .. code-block:: cmake
+
+      pybind11_precompile()
+      set_target_properties(pybind11_precompiled PROPERTIES
+                            INTERPROCEDURAL_OPTIMIZATION ON)
+
+* The library is static and per-build-tree; it is never installed or shared
+  between projects. Each extension module links its own copy, which keeps
+  pybind11's per-module state the same as in header-only mode.
+* Not available with ``PYBIND11_NOPYTHON`` (the library needs Python
+  headers).
+
+For build systems other than CMake, the same sources ship with the pybind11
+package: compile ``pybind11_combined.cpp`` from the directory reported by
+``python -m pybind11 --srcdir`` (also available as
+``pybind11.get_source_dir()`` and the ``srcdir`` pkg-config variable) into
+a static library or into your extension, and define
+``PYBIND11_PRECOMPILED`` for every translation unit.
+
+With Meson, build the library once per build tree and link it into each
+extension module, the same as the CMake path:
+
+.. code-block:: meson
+
+    pybind11_dep = dependency('pybind11')
+    pybind11_src = run_command(py, ['-m', 'pybind11', '--srcdir'],
+                               check : true).stdout().strip()
+
+    pybind11_precompiled = static_library('pybind11_precompiled',
+        pybind11_src / 'pybind11_combined.cpp',
+        cpp_args : ['-DPYBIND11_PRECOMPILED'],
+        gnu_symbol_visibility : 'hidden',
+        dependencies : [pybind11_dep, py.dependency()])
+
+    py.extension_module('example', 'example.cpp',
+        cpp_args : ['-DPYBIND11_PRECOMPILED'],
+        link_with : pybind11_precompiled,
+        dependencies : [pybind11_dep])
+
+The configuration-macro rules above apply here too: the static library and
+every module that links it must be compiled with the same configuration
+macros, and ``-DPYBIND11_PRECOMPILED`` must appear in both ``cpp_args``
+lists. (``pybind11_dep.get_variable('srcdir')`` also reports the source
+directory when Meson finds pybind11 through pkg-config.)
+
 Configuration variables
 -----------------------
 

docs/faq.rst

@@ -79,7 +79,12 @@ and the binding code
 How can I reduce the build time?
 ================================
 
-It's good practice to split binding code over multiple files, as in the
+First, consider the opt-in precompiled mode: it compiles the non-template
+part of pybind11 once per project instead of once for each translation unit.
+In CMake, this is one keyword on ``pybind11_add_module``. See
+:ref:`precompile-mode`.
+
+It's also good practice to split binding code over multiple files, as in the
 following example:
 
 :file:`example.cpp`:

include/pybind11/detail/internals.h

@@ -52,7 +52,8 @@
 // the library and the modules linking it. PYBIND11_MODULE calls it, so a mismatch (or a
 // missing library) surfaces as one readable undefined symbol at link time instead of many
 // unrelated ones at run time. If you add a configuration macro that changes the code in the
-// -inl.h files, encode it here and add it to the list in docs/compiling.rst.
+// -inl.h files, encode it here, add it to the list in docs/compiling.rst, and add it to
+// tools/check_inl_headers.py.
 #    if defined(Py_GIL_DISABLED)
 #        define PYBIND11_PRECOMPILED_CFG_GD 1
 #    else

tools/check_inl_headers.py

@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""
+Check the invariants of the -inl.h files used by the precompiled mode:
+
+* Every namespace-scope function definition is marked PYBIND11_INLINE.
+* Every macro in a preprocessor condition is known. A macro that changes the
+  code must either be the same for the library and the modules (platform,
+  compiler, Python version), or be encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK
+  (detail/internals.h) and listed in docs/compiling.rst.
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+# Same in the library and in the modules that link it.
+ENVIRONMENT_MACROS = {
+    "GRAALVM_PYTHON",
+    "NDEBUG",
+    "PYBIND11_BUILTIN_QUALNAME",
+    "PYBIND11_HAS_STRING_VIEW",
+    "PYBIND11_PRECOMPILED",
+    "PYPY_VERSION",
+    "PY_MAJOR_VERSION",
+    "PY_MINOR_VERSION",
+    "PY_VERSION_HEX",
+    "Py_REF_DEBUG",
+    "_MSC_VER",
+    "_WIN32",
+    "__GLIBCXX__",
+    "__GNUC__",
+    "__cpp_lib_unordered_map_try_emplace",
+    "__clang__",
+}
+
+# Encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK.
+GUARDED_MACROS = {
+    "PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET",
+    "PYBIND11_DETAILED_ERROR_MESSAGES",
+    "PYBIND11_HAS_SUBINTERPRETER_SUPPORT",
+    "PYBIND11_INTERNALS_VERSION",
+    "PYBIND11_SIMPLE_GIL_MANAGEMENT",
+    "Py_GIL_DISABLED",
+}
+
+# Only change code inside the library; documented in docs/compiling.rst.
+LIBRARY_ONLY_MACROS = {
+    "PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING",
+}
+
+KNOWN_MACROS = ENVIRONMENT_MACROS | GUARDED_MACROS | LIBRARY_ONLY_MACROS
+
+TOKENS = re.compile(
+    r"""
+    (?P<raw>R"(?P<delim>[^(\s]*)\(.*?\)(?P=delim)")
+    | (?P<str>"(?:\\.|[^"\\\n])*")
+    | (?P<chr>'(?:\\.|[^'\\\n])*')
+    | (?P<line_comment>//[^\n]*)
+    | (?P<block_comment>/\*.*?\*/)
+    """,
+    re.VERBOSE | re.DOTALL,
+)
+PP_CONDITION = re.compile(r"^[ \t]*#\s*(?:if|elif|ifdef|ifndef)\b(.*)$", re.MULTILINE)
+PP_LINE = re.compile(r"^[ \t]*#.*$", re.MULTILINE)
+NAMESPACE_MACRO = re.compile(
+    r"^[ \t]*PYBIND11_(?:NAMESPACE_BEGIN|NAMESPACE_END|WARNING_\w+)\(.*\)[ \t]*$",
+    re.MULTILINE,
+)
+IDENTIFIER = re.compile(r"\b[A-Za-z_]\w*\b")
+NON_FUNCTION_BLOCK = re.compile(r"^(?:struct|class|union|enum)\b|=$")
+
+
+def strip(text: str) -> str:
+    """Blank out comments and string literals, keeping the line numbers."""
+
+    def blank(match: re.Match[str]) -> str:
+        if match.group("line_comment") or match.group("block_comment"):
+            return re.sub(r"[^\n]", " ", match.group())
+        return '""' + "\n" * match.group().count("\n")
+
+    return TOKENS.sub(blank, text)
+
+
+def check(path: Path) -> list[str]:
+    text = strip(path.read_text(encoding="utf-8"))
+    errors: list[str] = []
+
+    for match in PP_CONDITION.finditer(text):
+        line = text.count("\n", 0, match.start()) + 1
+        errors.extend(
+            f"{path}:{line}: unknown configuration macro {name}; see {Path(__file__).name}"
+            for name in IDENTIFIER.findall(match.group(1))
+            if name != "defined" and name not in KNOWN_MACROS
+        )
+
+    code = PP_LINE.sub(lambda m: " " * len(m.group()), text)
+    code = NAMESPACE_MACRO.sub(lambda m: " " * len(m.group()), code)
+
+    # One entry per open brace: True for namespace and extern "C" blocks, whose
+    # contents are still at namespace scope.
+    scopes: list[bool] = []
+    start = 0
+    for pos, char in enumerate(code):
+        at_namespace_scope = all(scopes)
+        if char == "{":
+            if not at_namespace_scope:
+                scopes.append(False)
+                continue
+            header = " ".join(code[start:pos].split())
+            transparent = header == 'extern "C"' or header.startswith("namespace")
+            if (
+                not transparent
+                and "(" in header
+                and not NON_FUNCTION_BLOCK.search(header)
+                and "PYBIND11_INLINE" not in header.split()
+            ):
+                offset = start + len(code[start:pos]) - len(code[start:pos].lstrip())
+                line = code.count("\n", 0, offset) + 1
+                errors.append(f"{path}:{line}: definition without PYBIND11_INLINE")
+            scopes.append(transparent)
+            if transparent:
+                start = pos + 1
+        elif char == "}":
+            scopes.pop()
+            if all(scopes):
+                start = pos + 1
+        elif char == ";" and at_namespace_scope:
+            start = pos + 1
+
+    return errors
+
+
+def main(argv: list[str]) -> int:
+    errors = [error for arg in argv for error in check(Path(arg))]
+    for error in errors:
+        print(error)
+    return 1 if errors else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

tools/pybind11Config.cmake.in

@@ -18,6 +18,9 @@ This module sets the following variables in your project:
   Directories where pybind11 and python headers are located.
 ``pybind11_INCLUDE_DIR``
   Directory where pybind11 headers are located.
+``pybind11_SRC_DIR``
+  Directory where the library sources for the opt-in precompiled mode are
+  located (used by ``pybind11_precompile``).
 ``pybind11_DEFINITIONS``
   Definitions necessary to use pybind11, namely USING_pybind11.
 ``pybind11_LIBRARIES``
@@ -147,6 +150,7 @@ This module defines the following commands to assist with creating Python module
   pybind11_add_module(<target>
     [STATIC|SHARED|MODULE]
     [THIN_LTO] [OPT_SIZE] [NO_EXTRAS] [WITHOUT_SOABI]
+    [PRECOMPILE|NO_PRECOMPILE]
     <files>...
     )
 
@@ -162,6 +166,22 @@ default is ``MODULE``. There are several options:
   Disable the SOABI component (``PYBIND11_FINDPYTHON`` mode only).
 ``NO_EXTRAS``
   Disable all extras, exit immediately after making the module.
+``PRECOMPILE``
+  Link the target against the ``pybind11::precompiled`` static library
+  (created on first use); ``NO_PRECOMPILE`` opts a target out when the
+  ``PYBIND11_PRECOMPILE`` variable enables it globally.
+
+pybind11_precompile
+^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: cmake
+
+  pybind11_precompile()
+
+Create the ``pybind11::precompiled`` static library from the shipped sources
+(once per build tree). ``pybind11_add_module(... PRECOMPILE)`` calls this for
+you; call it directly to link ``pybind11::precompiled`` into your own
+targets.
 
 pybind11_strip
 ^^^^^^^^^^^^^^