diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 62511523de..d61e642f7a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -141,6 +141,11 @@ repos: language: pygrep entry: PyBind|\bNumpy\b|Cmake|CCache|PyTest exclude: ^\.pre-commit-config.yaml$ + - id: check-inl-headers + name: Check -inl.h files for the precompiled mode + language: python + entry: python tools/check_inl_headers.py + files: -inl\.h$ # PyLint has native support - not always usable, but works for us - repo: https://github.com/PyCQA/pylint diff --git a/docs/compiling.rst b/docs/compiling.rst index a6bee86ffe..a81b16fead 100644 --- a/docs/compiling.rst +++ b/docs/compiling.rst @@ -348,7 +348,8 @@ function with the following signature: .. code-block:: cmake pybind11_add_module( [MODULE | SHARED] [EXCLUDE_FROM_ALL] - [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] source1 [source2 ...]) + [NO_EXTRAS] [THIN_LTO] [OPT_SIZE] [PRECOMPILE | NO_PRECOMPILE] + source1 [source2 ...]) This function behaves very much like CMake's builtin ``add_library`` (in fact, it's a wrapper function around that command). It will add a library target @@ -404,6 +405,104 @@ optimizations remain disabled. .. _ThinLTO: http://clang.llvm.org/docs/ThinLTO.html +.. _precompile-mode: + +Pre-compiling part of pybind11 +------------------------------ + +pybind11 is header-only by default: every translation unit compiles its own +copy of the non-template implementation. The opt-in *precompiled* mode +compiles that implementation once, into a static library built inside your +own project with your own flags. This reduces the build time, most of all for +projects with many translation units or many modules in one build. + +.. code-block:: cmake + + pybind11_add_module(example PRECOMPILE example.cpp) + +The first ``PRECOMPILE`` target creates the library target +``pybind11::precompiled``; further targets reuse it. Set the CMake variable +``PYBIND11_PRECOMPILE`` to make it the default for all +``pybind11_add_module`` calls; use ``NO_PRECOMPILE`` on a target to opt back +out. For targets you create yourself, call the ``pybind11_precompile()`` +function and link ``pybind11::precompiled`` PRIVATE; the target carries the +required ``PYBIND11_PRECOMPILED`` compile definition PUBLIC, so your sources +also get it. + +Requirements and caveats: + +* The library and every module linking it must agree on the configuration + macros ``PYBIND11_INTERNALS_VERSION``, ``Py_GIL_DISABLED``, + ``PYBIND11_SIMPLE_GIL_MANAGEMENT``, + ``PYBIND11_DETAILED_ERROR_MESSAGES`` (defaults on in debug builds), + ``PYBIND11_HAS_SUBINTERPRETER_SUPPORT``, and + ``PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET``. A mismatch produces one + readable undefined symbol at link time referencing + ``pybind11_precompiled_config``. +* Configuration macros that only change code inside the library (for example + ``PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING``) must be defined when the + library is compiled; a definition only on your module has no effect. This + also applies to a custom ``PYBIND11_NAMESPACE`` visibility attribute: set + it at the directory level (``add_compile_definitions``) so that the library + gets it too. +* The library picks up your directory-level flags and C++ standard when it is + first created, so set those before the first ``PRECOMPILE`` target. A + status message reports the directory that created the library. +* The library is not compiled with link-time optimization, and the per-target + ``THIN_LTO`` and ``OPT_SIZE`` options of ``pybind11_add_module`` do not + apply to it. In a Release build this adds a few nanoseconds to each call of + a bound function (up to about 10% for a function that does nothing, + depending on the compiler; a few percent with LTO on the library). To change this, call ``pybind11_precompile()`` + yourself and + set the properties on the created target, ``pybind11_precompiled`` (the + real target behind the ``pybind11::precompiled`` alias; CMake does not let + you set properties through an alias): + + .. code-block:: cmake + + pybind11_precompile() + set_target_properties(pybind11_precompiled PROPERTIES + INTERPROCEDURAL_OPTIMIZATION ON) + +* The library is static and per-build-tree; it is never installed or shared + between projects. Each extension module links its own copy, which keeps + pybind11's per-module state the same as in header-only mode. +* Not available with ``PYBIND11_NOPYTHON`` (the library needs Python + headers). + +For build systems other than CMake, the same sources ship with the pybind11 +package: compile ``pybind11_combined.cpp`` from the directory reported by +``python -m pybind11 --srcdir`` (also available as +``pybind11.get_source_dir()`` and the ``srcdir`` pkg-config variable) into +a static library or into your extension, and define +``PYBIND11_PRECOMPILED`` for every translation unit. + +With Meson, build the library once per build tree and link it into each +extension module, the same as the CMake path: + +.. code-block:: meson + + pybind11_dep = dependency('pybind11') + pybind11_src = run_command(py, ['-m', 'pybind11', '--srcdir'], + check : true).stdout().strip() + + pybind11_precompiled = static_library('pybind11_precompiled', + pybind11_src / 'pybind11_combined.cpp', + cpp_args : ['-DPYBIND11_PRECOMPILED'], + gnu_symbol_visibility : 'hidden', + dependencies : [pybind11_dep, py.dependency()]) + + py.extension_module('example', 'example.cpp', + cpp_args : ['-DPYBIND11_PRECOMPILED'], + link_with : pybind11_precompiled, + dependencies : [pybind11_dep]) + +The configuration-macro rules above apply here too: the static library and +every module that links it must be compiled with the same configuration +macros, and ``-DPYBIND11_PRECOMPILED`` must appear in both ``cpp_args`` +lists. (``pybind11_dep.get_variable('srcdir')`` also reports the source +directory when Meson finds pybind11 through pkg-config.) + Configuration variables ----------------------- diff --git a/docs/faq.rst b/docs/faq.rst index 2b89d203f3..8a061c812d 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -79,7 +79,12 @@ and the binding code How can I reduce the build time? ================================ -It's good practice to split binding code over multiple files, as in the +First, consider the opt-in precompiled mode: it compiles the non-template +part of pybind11 once per project instead of once for each translation unit. +In CMake, this is one keyword on ``pybind11_add_module``. See +:ref:`precompile-mode`. + +It's also good practice to split binding code over multiple files, as in the following example: :file:`example.cpp`: diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 224198ce3d..164da39b5c 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -52,7 +52,8 @@ // the library and the modules linking it. PYBIND11_MODULE calls it, so a mismatch (or a // missing library) surfaces as one readable undefined symbol at link time instead of many // unrelated ones at run time. If you add a configuration macro that changes the code in the -// -inl.h files, encode it here and add it to the list in docs/compiling.rst. +// -inl.h files, encode it here, add it to the list in docs/compiling.rst, and add it to +// tools/check_inl_headers.py. # if defined(Py_GIL_DISABLED) # define PYBIND11_PRECOMPILED_CFG_GD 1 # else diff --git a/tools/check_inl_headers.py b/tools/check_inl_headers.py new file mode 100755 index 0000000000..6787fb740b --- /dev/null +++ b/tools/check_inl_headers.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +""" +Check the invariants of the -inl.h files used by the precompiled mode: + +* Every namespace-scope function definition is marked PYBIND11_INLINE. +* Every macro in a preprocessor condition is known. A macro that changes the + code must either be the same for the library and the modules (platform, + compiler, Python version), or be encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK + (detail/internals.h) and listed in docs/compiling.rst. +""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path + +# Same in the library and in the modules that link it. +ENVIRONMENT_MACROS = { + "GRAALVM_PYTHON", + "NDEBUG", + "PYBIND11_BUILTIN_QUALNAME", + "PYBIND11_HAS_STRING_VIEW", + "PYBIND11_PRECOMPILED", + "PYPY_VERSION", + "PY_MAJOR_VERSION", + "PY_MINOR_VERSION", + "PY_VERSION_HEX", + "Py_REF_DEBUG", + "_MSC_VER", + "_WIN32", + "__GLIBCXX__", + "__GNUC__", + "__cpp_lib_unordered_map_try_emplace", + "__clang__", +} + +# Encoded in PYBIND11_PRECOMPILED_CONFIG_CHECK. +GUARDED_MACROS = { + "PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET", + "PYBIND11_DETAILED_ERROR_MESSAGES", + "PYBIND11_HAS_SUBINTERPRETER_SUPPORT", + "PYBIND11_INTERNALS_VERSION", + "PYBIND11_SIMPLE_GIL_MANAGEMENT", + "Py_GIL_DISABLED", +} + +# Only change code inside the library; documented in docs/compiling.rst. +LIBRARY_ONLY_MACROS = { + "PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING", +} + +KNOWN_MACROS = ENVIRONMENT_MACROS | GUARDED_MACROS | LIBRARY_ONLY_MACROS + +TOKENS = re.compile( + r""" + (?PR"(?P[^(\s]*)\(.*?\)(?P=delim)") + | (?P"(?:\\.|[^"\\\n])*") + | (?P'(?:\\.|[^'\\\n])*') + | (?P//[^\n]*) + | (?P/\*.*?\*/) + """, + re.VERBOSE | re.DOTALL, +) +PP_CONDITION = re.compile(r"^[ \t]*#\s*(?:if|elif|ifdef|ifndef)\b(.*)$", re.MULTILINE) +PP_LINE = re.compile(r"^[ \t]*#.*$", re.MULTILINE) +NAMESPACE_MACRO = re.compile( + r"^[ \t]*PYBIND11_(?:NAMESPACE_BEGIN|NAMESPACE_END|WARNING_\w+)\(.*\)[ \t]*$", + re.MULTILINE, +) +IDENTIFIER = re.compile(r"\b[A-Za-z_]\w*\b") +NON_FUNCTION_BLOCK = re.compile(r"^(?:struct|class|union|enum)\b|=$") + + +def strip(text: str) -> str: + """Blank out comments and string literals, keeping the line numbers.""" + + def blank(match: re.Match[str]) -> str: + if match.group("line_comment") or match.group("block_comment"): + return re.sub(r"[^\n]", " ", match.group()) + return '""' + "\n" * match.group().count("\n") + + return TOKENS.sub(blank, text) + + +def check(path: Path) -> list[str]: + text = strip(path.read_text(encoding="utf-8")) + errors: list[str] = [] + + for match in PP_CONDITION.finditer(text): + line = text.count("\n", 0, match.start()) + 1 + errors.extend( + f"{path}:{line}: unknown configuration macro {name}; see {Path(__file__).name}" + for name in IDENTIFIER.findall(match.group(1)) + if name != "defined" and name not in KNOWN_MACROS + ) + + code = PP_LINE.sub(lambda m: " " * len(m.group()), text) + code = NAMESPACE_MACRO.sub(lambda m: " " * len(m.group()), code) + + # One entry per open brace: True for namespace and extern "C" blocks, whose + # contents are still at namespace scope. + scopes: list[bool] = [] + start = 0 + for pos, char in enumerate(code): + at_namespace_scope = all(scopes) + if char == "{": + if not at_namespace_scope: + scopes.append(False) + continue + header = " ".join(code[start:pos].split()) + transparent = header == 'extern "C"' or header.startswith("namespace") + if ( + not transparent + and "(" in header + and not NON_FUNCTION_BLOCK.search(header) + and "PYBIND11_INLINE" not in header.split() + ): + offset = start + len(code[start:pos]) - len(code[start:pos].lstrip()) + line = code.count("\n", 0, offset) + 1 + errors.append(f"{path}:{line}: definition without PYBIND11_INLINE") + scopes.append(transparent) + if transparent: + start = pos + 1 + elif char == "}": + scopes.pop() + if all(scopes): + start = pos + 1 + elif char == ";" and at_namespace_scope: + start = pos + 1 + + return errors + + +def main(argv: list[str]) -> int: + errors = [error for arg in argv for error in check(Path(arg))] + for error in errors: + print(error) + return 1 if errors else 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/tools/pybind11Config.cmake.in b/tools/pybind11Config.cmake.in index abcd43e199..d666e9cc96 100644 --- a/tools/pybind11Config.cmake.in +++ b/tools/pybind11Config.cmake.in @@ -18,6 +18,9 @@ This module sets the following variables in your project: Directories where pybind11 and python headers are located. ``pybind11_INCLUDE_DIR`` Directory where pybind11 headers are located. +``pybind11_SRC_DIR`` + Directory where the library sources for the opt-in precompiled mode are + located (used by ``pybind11_precompile``). ``pybind11_DEFINITIONS`` Definitions necessary to use pybind11, namely USING_pybind11. ``pybind11_LIBRARIES`` @@ -147,6 +150,7 @@ This module defines the following commands to assist with creating Python module pybind11_add_module( [STATIC|SHARED|MODULE] [THIN_LTO] [OPT_SIZE] [NO_EXTRAS] [WITHOUT_SOABI] + [PRECOMPILE|NO_PRECOMPILE] ... ) @@ -162,6 +166,22 @@ default is ``MODULE``. There are several options: Disable the SOABI component (``PYBIND11_FINDPYTHON`` mode only). ``NO_EXTRAS`` Disable all extras, exit immediately after making the module. +``PRECOMPILE`` + Link the target against the ``pybind11::precompiled`` static library + (created on first use); ``NO_PRECOMPILE`` opts a target out when the + ``PYBIND11_PRECOMPILE`` variable enables it globally. + +pybind11_precompile +^^^^^^^^^^^^^^^^^^^ + +.. code-block:: cmake + + pybind11_precompile() + +Create the ``pybind11::precompiled`` static library from the shipped sources +(once per build tree). ``pybind11_add_module(... PRECOMPILE)`` calls this for +you; call it directly to link ``pybind11::precompiled`` into your own +targets. pybind11_strip ^^^^^^^^^^^^^^