Generalize the ABI provider into a generic "reserved provider", v2

peps/pep-9999.rst

@@ -82,23 +82,30 @@ governed by a *variant provider*. Variant providers supply ordered lists
 of compatible features and feature values corresponding to their
 namespaces, as required by :pep:`825`.
 
-The namespace ``abi_dependency`` is reserved for the `ABI Dependency
-Variant Provider <#abi-dependency-variant-provider-optional>`_. The
-providers for all the other namespaces used MUST be defined in the
-`provider information`_ dictionary in `variant metadata`_. This PEP is
-concerned only with consuming said dictionary; the process of building
-variant wheels will be covered in a subsequent PEP.
+The providers for all namespaces used MUST be defined in the `provider
+information`_ dictionary in `variant metadata`_. This PEP is concerned
+only with consuming said dictionary; the process of building variant
+wheels will be covered in a subsequent PEP.
 
 Each of these providers either defines a static list of compatible
-features and their values, or specifies a list of requirements referring
-to Python packages. In the latter case, an API endpoint is either
-explicitly specified or inferred from the first dependency string. The
-canonical way of querying the list of compatible features and their
-values is then to install the specified packages and call this endpoint.
-The combination of dependency strings and the API endpoint are used to
-uniquely identify the provider, while the respective version constraints
-can be used to indicate the minimum versions emitting variant properties
-used in the package.
+features and their values, specifies a list of requirements referring
+to Python packages or is one of the `builtin providers`_.
+
+Builtin providers are implemented within the installer to handle checks
+that the plugin API cannot support. For example, the `ABI Dependency
+Variant Provider`_ checks whether a wheel built against a particular
+version of a dependency (e.g. PyTorch) is compatible with the version
+selected during dependency resolution. This requires access to the
+installer's dependency resolver.
+
+For Python package providers (also called provider plugins), an API
+endpoint is either explicitly specified or inferred from the first
+dependency string. The canonical way of querying the list of compatible
+features and their values is then to install the specified packages and
+call this endpoint. The combination of dependency strings and the API
+endpoint are used to uniquely identify the provider, while the
+respective version constraints can be used to indicate the minimum
+versions emitting variant properties used in the package.
 
 For security reasons, none of these packages are installed or used by
 default. Tools can provide a way to securely use a subset of provider
@@ -120,7 +127,7 @@ namespace:
 1. The tool SHOULD implement a way for the user to provide a static list
    of compatible features and their values for a particular provider.
    Standardizing this format is left to a future PEP. If the user
-   provided said list, the tool MUST use it exclusively.  Otherwise,
+   provided said list, the tool MUST use it exclusively. Otherwise,
    proceed to step 2.
 
 2. The tool MUST read the ``optional`` key from the `provider
@@ -129,24 +136,29 @@ namespace:
    and MAY provide a way to disable one. If a provider is disabled, the
    list of compatible features is empty. Otherwise, proceed to step 3.
 
-3. The tool MUST read the ``static-properties`` key from the `provider
+3. The tool MUST read the ``builtin`` key from the `provider
+   information`_ dictionary. If it is present, the tool MUST use the
+   internal provider implementation if supported, or assume that no
+   features are compatible if not. Otherwise, proceed to step 4.
+
+4. The tool MUST read the ``static-properties`` key from the `provider
    information`_ dictionary. If it is present, the tool MUST use the
    static lists of properties therein exclusively. Otherwise, proceed to
-   step 4.
+   step 5.
 
-4. If no static list of compatible features and their values is
+5. If no static list of compatible features and their values is
    provided, the tool MUST read the ``requires`` and ``plugin-api`` keys
    in the `provider information`_ dictionary to determine the packages
    providing the compatibility information. If the tool provides a
    secure implementation of the specific providers, it SHOULD obtain
-   the list from their implementation. Otherwise, proceed to step 5.
+   the list from their implementation. Otherwise, proceed to step 6.
 
-5. The tool SHOULD provide a way for the user to permit installing
+6. The tool SHOULD provide a way for the user to permit installing
    specific provider packages. It MUST NOT install or use any provider
    packages that are not considered trusted. It SHOULD use an isolated
    virtual environment while installing provider plugin packages.
 
-6. The tool MUST query the trusted and installed provider packages via
+7. The tool MUST query the trusted and installed provider packages via
    the `provider plugin API`_ to obtain the list of compatible features
    and feature values. For providers that were not trusted, it MUST
    assume that the list of compatible features is empty.
@@ -172,12 +184,13 @@ subsection.
     +- variants
     +- providers
        +- {namespace}
-          +- feature-order     : list[str]  = []
+          +- builtin           : str
+          +- feature-order     : list[str]
           +- optional          : bool       = False
-          +- plugin-api        : str | None = None
-          +- requires          : list[str]  = []
+          +- plugin-api        : str
+          +- requires          : list[str]
           +- static-properties
-             +- {feature}      : list[str]  = []
+             +- {feature}      : list[str]
 
 This structure corresponds to the version ``0.2.0`` of the format. An
 update of the proposed JSON schema for the current format version is
@@ -219,6 +232,12 @@ information dictionary:
   dependency that remains after filtering MUST always reference the same
   package name as the first dependency prior to filtering.
 
+- ``builtin: str``: Specifies the unique name identifying the builtin
+  provider specified in a PEP. If it is provided, the provider is
+  identified by the value and MUST behave as specified in the PEP
+  defining it. If the tool does not recognize the particular provider,
+  it MUST assume that its list of compatible features is empty.
+
 A provider information dictionary MAY additionally contain the following
 key:
 
@@ -237,9 +256,6 @@ contain the following key:
   ``requires`` and then replacing all ``-`` characters with ``_`` in the
   normalized package name.
 
-It is invalid to specify ``requires`` or ``plugin-api`` if
-``static-properties`` are present.
-
 If the ``static-properties`` key is present, the dictionary MAY
 additionally contain the following key:
 
@@ -248,6 +264,13 @@ additionally contain the following key:
   ``static-properties`` MUST be listed here. Entries not found in the
   ``static-properties`` dictionary MUST NOT appear in this list.
 
+To reiterate, in addition to the ``optional`` key, the following key
+combinations are valid:
+
+- ``requires``, with optional ``plugin-api``
+- ``static-properties``, with optional ``feature-order``
+- ``builtin``
+
 The ``providers`` dictionary is copied into index-level metadata. For
 the metadata to be consistent, the same keys MUST always correspond to
 the same values. When combining metadata, the resulting ``providers``
@@ -268,7 +291,7 @@ from :pep:`825` can be extended in the following way:
 
       "default-priorities": {
         // MUST list all namespaces used.
-        "namespace": ["x86_64", "aarch64", "blas_lapack"],
+        "namespace": ["x86_64", "aarch64", "blas_lapack", "abi"],
       },
 
       "providers": {
@@ -280,6 +303,9 @@ from :pep:`825` can be extended in the following way:
           // "plugin-api" is OPTIONAL here. It is inferred from "requires":
           // "plugin-api": "provider_variant_aarch64"
         },
+        "abi": {
+          "builtin": "abi_dependency"
+        },
         "blas_lapack": {
           // Specifies compatible properties. No install-time querying is
           // necessary.
@@ -319,11 +345,11 @@ published in the same indexes as the packages using them. In particular,
 packages published to PyPI MUST NOT rely on plugins that need to be
 installed from other indexes.
 
-Except for namespaces reserved as part of this PEP and variant providers
-using the ``static-properties`` key, installable Python packages MUST be
-provided for plugins. The entire dependency tree of such packages MUST
-be installable using non-variant wheels, and variant wheels MUST NOT be
-used while installing them.
+Except for variant providers using the ``static-properties`` and
+``builtin`` keys, installable Python packages MUST be provided for
+plugins. The entire dependency tree of such packages MUST be installable
+using non-variant wheels, and variant wheels MUST NOT be used while
+installing them.
 
 As noted in the `Providers`_ section, these plugins can also be
 reimplemented by tools needing them. If that is the case, the API
@@ -387,7 +413,7 @@ Variant plugin discovery
 ''''''''''''''''''''''''
 
 The Python packages providing variant provider plugins SHOULD install an
-entry point to facilitate discovery by variant-related utilities.  The
+entry point to facilitate discovery by variant-related utilities. The
 entry point MUST be placed in the ``variant_plugins`` group, its name
 being the recommended namespace name, and its value being the plugin's
 `API endpoint`_. For example, a plugin can declare the following entry
@@ -567,24 +593,43 @@ an underscore (``_``) character to avoid incidental conflicts with
 future extensions.
 
 
-ABI Dependency Variant Provider (Optional)
--------------------------------------------
+Builtin providers
+-----------------
+
+Some compatibility checks require integration with a tool's internals,
+such as its dependency resolver, that the plugin API does not expose.
+Builtin providers handle these checks directly within the tool. They
+are defined in PEPs, and their identifiers are used in the ``builtin``
+key in the `provider information`_ dictionary.
+
+All builtin providers are OPTIONAL. Tools that choose to implement them
+MUST follow the PEP implementing them. Tools that do not recognize a
+particular builtin provider MUST treat the variants using it as
+incompatible, and SHOULD inform users when such wheels are skipped.
+
+The following builtin variant providers are defined:
+
+================== ==================================
+Identifier         Provider
+================== ==================================
+``abi_dependency`` `ABI Dependency Variant Provider`_
+================== ==================================
+
+Subsequent PEPs may define additional builtin providers.
+
 
-This section describes an OPTIONAL extension to the wheel variant
-specification. Tools that choose to implement this feature MUST follow
-this specification. Tools that do not implement this feature MUST treat
-the variants using it as incompatible, and SHOULD inform users when such
-wheels are skipped.
+ABI Dependency Variant Provider
+'''''''''''''''''''''''''''''''
 
-The variant namespace ``abi_dependency`` is reserved for expressing that
-different builds of the same version of a package are compatible with
-different versions or version ranges of a dependency. This namespace
-MUST NOT be listed in the `provider information`_ dictionary nor in the
-``default-priorities.namespace`` list, and can only appear in the
-``variants`` dictionary. It is not taken into consideration in variant
-ordering. Package maintainers SHOULD NOT be publishing variant wheels
-where two different ``abi_dependency`` property sets could be compatible
-with the same system.
+The ABI Dependency Variant Provider is used to express that different
+builds of the same version of a package are compatible with different
+versions or version ranges of a dependency. Its identifier is
+``abi_dependency``. The ordering between features and feature values is
+undefined, and the respective namespace should be listed after all the
+ordering-significant namespaces in the ``default-priorities.namespace``
+list. Package maintainers SHOULD NOT be publishing variant wheels where
+two different ABI dependency property sets could be compatible with
+the same system.
 
 Within this namespace, zero or more properties can be used to express
 compatible dependency versions. For each property, the feature name MUST
@@ -695,13 +740,14 @@ discovery when installed in development environments. This can be used
 by variant-related tools, for example to aid debugging variant selection
 or help configuring source trees.
 
-The `ABI Dependency Variant Provider
-<#abi-dependency-variant-provider-optional>`_ is defined separately, as
+The `ABI Dependency Variant Provider`_ is defined separately, as
 it needs to interact with the dependency resolver. To avoid adding
 significant complexity to the plugin API and at the same time
 restricting the actual implementation, it has been made a special case.
 It is entirely optional to avoid adding maintenance burden to tool
-maintainers.
+maintainers. At the same time, provisions were made to permit creating
+further `builtin providers`_ in the future, and to keep the provider
+information consistent between all types of providers.
 
 
 Backwards Compatibility
@@ -820,12 +866,16 @@ Change History
     required to be non-empty.
   - The entry point name is now used to provide the recommended
     namespace for a provider.
-  - The feature names and values in the `ABI Dependency Variant Provider
-    (Optional)`_ are now normalized according to the wheel normalization
+  - The feature names and values in the `ABI Dependency Variant
+    Provider`_ are now normalized according to the wheel normalization
     rules, to match the restrictions in :pep:`825`.
   - A provision has been added that a consistent list of compatible
     feature names and values from a single provider must be used
     throughout the install session.
+  - The `ABI Dependency Variant Provider`_ now requires an explicit
+    entry in the ``default-priorities.namespace`` and ``providers``
+    keys, and it was generalized into one of the possible `builtin
+    providers`_.
 
 
 Appendices

peps/pep-9999/variant-schema-0.2.0.json

@@ -37,6 +37,11 @@
           "type": "object",
           "description": "Provider information",
           "properties": {
+            "builtin": {
+              "description": "The identifier of a builtin provider if this is one",
+              "type": "string",
+              "enum": ["abi_dependency"]
+            },
             "feature-order": {
               "description": "Feature order for static-properties",
               "type": "array",
@@ -86,10 +91,22 @@
           },
           "additionalProperties": false,
           "oneOf": [
+            {
+              "required": ["builtin"],
+              "not": {
+                "anyOf": [
+                  { "required": ["feature-order"] },
+                  { "required": ["plugin-api"] },
+                  { "required": ["requires"] },
+                  { "required": ["static-properties"] }
+                ]
+              }
+            },
             {
               "required": ["requires"],
               "not": {
                 "anyOf": [
+                  { "required": ["builtin"] },
                   { "required": ["feature-order"] },
                   { "required": ["static-properties"] }
                 ]
@@ -99,6 +116,7 @@
               "required": ["static-properties"],
               "not": {
                 "anyOf": [
+                  { "required": ["builtin"] },
                   { "required": ["requires"] },
                   { "required": ["plugin-api"] }
                 ]