PEP xxxx: Providers [rebased]

[mgorny]

Unfortunately, I didn't notice that #40 was based on the pep-wheel-variants-acceptance branch before deleting it, so it ended up closing with no option to reopen it. I'm opening a new thread to continue working on it. I think all the comments from the previous thread have been addressed, and I've rebased it on the current PEP 825 state (per the integration branch). I still need to sync it to the changes in 825.

[read-the-docs-community]

Documentation build overview

📚 wheelnext-peps | 🛠️ Build #34678495 | 📁 Comparing 536fabf against latest (0cde95d)

  🔍 Preview build  

767 files changed · + 18 added · ± 749 modified

+ Added

• pep-0805/index.html
• pep-0835/index.html
• pep-0836/index.html
• pep-0837/index.html
• pep-0838/index.html
• pep-0839/index.html
• pep-0840/index.html
• pep-0841/index.html
• pep-0842/index.html
• pep-0843/index.html
• and 8 more...

± Modified

• 404.html
• index.html
• api/index.html
• contents/index.html
• docs/index.html
• numerical/index.html
• pep-0000/index.html
• pep-0001/index.html
• pep-0002/index.html
• pep-0003/index.html
• and 739 more...

[mgorny]

Updated the schema, added consistency notes, and ready for another round of review.

[mgorny]

Okay, I'm done rereading and updating it.

[rgommers]

Something I just noticed: the text contains only an example about the index-level variants.json, and doesn't say explicitly that the provider metadata needs to be added to .dist-info/variant.json - it should.

[konsti-openai]

I have a lot more material for this section from my EuroPython talk, if someone really wants hardcoded providers I can write something up, but right now this should be enough.

[mgorny]

Okay, I'll squash, rebase and open a PR to the peps repo. Stay tuned!

[rgommers]

I made this a MUST in PEP 817, and I believe that that is right. It cannot really be optional for installers.

[konstin]

I don't think every installer needs to provide this: Imagine a intentionally minimal tool such as flit, or a company internal tool. Both get the user what they want and need, they act correctly on the protocol level. If we say MUST, we'd consider a tool that installs the correct variant wheel as wrong for missing this feature.

[rgommers]

I think that's going to run into strong opposition, and also clashes with use cases we accepted like "don't install for the current machine".

What I have primarily in mind though is uv/pip: installers that do resolution and aim to implement all packaging standards. I think for minimal implementations like flit/installer, I think the likelihood of implementing anything is, from high to low:

• Nothing beyond direct <tool> install a_variant_wheel.whl
• A static file format
• Support for provider with install-time behavior

Stating that the more dynamic behavior is MUST while the "let the user determine it" is SHOULD seems wrong, especially when people are primarily going to be reading this with pip/uv in mind.

[konstin]

I believe that we get the right level with SHOULD: It's required to create the ideal experience that covers all of our use cases, but it doesn't break interoperability if you don't implement it. IMO, it would be even better to move it out of the spec section entirely and move it to a separate, non-normative section "Intended User Experience and Tool Suggestions".

As I understand it, a MUST is for something that would break interoperability or has otherwise unacceptable consequences (such as security problems). Everything else can be a SHOULD or MAY requirement: The PEP doesn't require good or even useful tools, it only requires what we need that different parts in the ecosystem can talk to each other and that certain expectations are met (e.g.: If I publish a wheel with the GPU provider, my users actually get the GPU wheel). My stance on the scope of a PEP is that it describes a problem, specifies a set of core rules, and then explains how those rules solve that problem if people build on top of them. This assumes good faith and that tool authors are experts.

A static input file isn't something that we need to get a working variant-enabled installation (and can still do a PEP for it later - I don't disagree that this will be the most popular feature for tools). I'd argue that in Python packaging, we leave the majority of scope of all features and design to the tools themselves, and that we're better off for it! It allows experimentation, evolution and competition between tools. If we write something as a MUST now, it's locked except for an arduous change process, which is a pain for tools where we got it wrong (and from experience python is so diverse, user feedback usually proves you wrong, or otherwise a few years later the best practices and requirements have changed). We should insist on the separation between interface proposal and tool behavior and avoid micromanaging tool maintainers.

[rgommers]

or has otherwise unacceptable consequences (such as security problems).

This is exactly what some people will argue is the consequence. "I don't want to run third-party code, hence I must have an override to be able to install this package at all".

EDIT: or, aside from security, "I need to build a portable container image".

A static input file isn't something that we need [...] We should insist on the separation between interface proposal and tool behavior and avoid micromanaging tool maintainers.

I don't think we're disagreeing here? I am not saying we need to standardize the file format or any specific UX, only that having some kind of UX (can be tool-specific) for specifying deterministically what the required features/values are for a given package install is a must-have.

[rgommers]

I think if you read the new The Design Space section of my PEP 817 PR, you will see what the gap is here between this SHOULD and what's written there.

[mgorny]

I'm with @konstin here. I don't think we should enforce support for features that may be absolutely useless from the particular tool's perspective, or force tool authors to deliberately stray from the specification. SHOULD is a strong enough word, given that per definition it means "you have to have a really good reason not to do that".

[rgommers]

Thanks @mgorny and @konsti-openai. I went through after finishing with 817 updates, and this looks quite good. Added some minor comments inline. The only two comments I have that are slightly more substantial, and I think deserve adding a bit of content for, are:

• No error behavior, need to say that if a provider errors in some way, the install process has to fail rather than silently change the answer.
• It's not stated that caching the results from querying a provider is allowed. In 817 we had "Clarified that the value returned by get_supported_configs() may be cached." in Change History.

[mgorny]

I've fixed the trivial issues. Waiting with the bigger changes until the discussion is resolved.