RFC: Extended Standard Library (ESL)

[bdaehlie]

This RFC describes an Extended Standard Library (ESL) for Rust.

Please see this document for additional information about fundraising.

I’d like to thank Dirkjan Ochtman (@djc), Adolfo Ochagavía (@aochagavia), Alex Gaynor (@alex), James Munns (OneVariable GmbH, @jamesmunns), Andrew Gallant (@BurntSushi), Marco Ieni (@marcoieni), and many others for their feedback and assistance while working on this. My thanks does not necessarily imply their endorsement.

Rendered

[Manishearth]

implementation effort is ongoing. I don't think this ideal is difficult at all.

I'd argue that rustlang::foobar (or rust_esl::foobar) for ESL crates is a pretty good use of that feature, and is a good place to put funding if things need to be expedited.

I don't see another easy way to have this be clear from the names in a way that's not easily spoofed.

[bdaehlie]

If we could pull it off that would be ideal. It's clearly better. But it depends on namespacing being implemented and having the relevant maintainers on board with renames.

[Manishearth]

It seems like renames are necessary here anyway? Maybe I missed something but I think a very important facet that isn't being really discussed in depth this RFC is how ESL membership is signified.

[bdaehlie]

Right now the RFC does not require renaming crates, only moving them to the ESL GitHub organization:

"As such, pre-existing crates that join the ESL will keep their names unless there is a compelling reason to make a change" ... "Crate users will have to confirm ESL membership by checking the GitHub organization."

The ability to confirm membership is important. Originally my draft RFC did require RFC 3243 implementation and renames into an ESL namespace in order to provide clarity about functionality and membership. Concerns were raised about the difficulty of renaming existing popular crates, so this compromise was made. If the community thought renaming crates into an ESL namespace was feasible that would be a better outcome and I'd be happy to update the RFC as such. I'd want to hear from potential ESL crate maintainers first though.

[bdaehlie]

Also, I don't know if you got there yet, but the Naming and Namespaces section of "Future Possibilities" further discusses crate naming and indicating membership.

[Manishearth]

The ability to confirm membership is important.

I think this RFC would be more compelling if it spent significant time talking about this: what the options are, what the "minimum viable" thing is for this RFC to be considered implemented.

In its current status "make a github organization" doesn't really feel like a compelling RFC for the Rust project. I understand that there are no readymade solutions on the table available right now, but that doesn't mean you can't talk about what the solutions could be. If this RFC needs additional cargo features to make sense, so be it.

It seems like the primary goal of this RFC is trust, and it spends a surprisingly small amount of time talking about mechanisms for achieving trust. It talks about what makes up a trustworthy crate (good!!), but being trustworthy does not make something trusted: how do you plan to make it easy for developers to trust these crates? Goal #2 explicitly talks about there being a subpar experience on choosing crates, but I don't see how this really helps that developer experience along.

Concerns were raised about the difficulty of renaming existing popular crates

"Renames as redirects" is a thing that could be proposed. Indeed, we were considering proposing it as a part of the namespaces project before deciding to defer that for later.

[bdaehlie]

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

The concern pointed out to me about renames was not so much about literally renaming or redirecting crates, but about the volume of extant documentation, both formal and information (e.g. blog posts, stack overflow, that sort of stuff).

[epage]

This is organizational namespacing which goes against the spirit of that RFC.

I was going to raise a separate topic that naming should be out of scope.

[Manishearth]

@epage Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

My point here is less about reintroducing the namespace requirement and more about bringing more focus onto how we make this trusted relationship clear to users. There are many mechanisms to do so, namespacing is one of them.

[epage]

Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

There are trade offs to using this outside of its intended purpose. Our example of doing so will be noticed much more than any caveats we put in the documentation around this feature.

[ChrisDenton]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

[JohnDowson]

Some, perhaps, even a language feature. Bitfields probably being the least controversial candidate.

[ssokolow]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

As someone who's been here since about 2013, it's been my observation that Tranche 1 stuff that isn't in the standard library had very good reason to not be in the standard library back then, and the question is whether they've managed to stabilize their APIs enough in the intervening 12 years for that to no longer be an issue.

I can literally name, off the top of my head, at least one (sometimes two or more) formerly-leading implementations of every item on the latter half of that list which were, at the time, the leading solution:

• Random number generation (rand has gone through multiple API breaks)
• Serialization/deserialization (Serde supplanted rustc_serialize, which was once the leading solution)
• Data structure serialization (Same thing. We'd have been stuck with rustc_serialize if we moved earlier.)
• Date/time support (We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.)
  Error handling support (error-chain to failure, failure to anyhow and thiserror, with eyre around 1/7th of anyhow's popularity and climbing... and that doesn't count the ecosystem's "Yeah, no." reaction to a team member's idea of what a good error-handling API should look like in fehler.)

It may be slow, but things do make it into std in good time, as demonstrated by the incorporation of a variation on once-cell as it became apparent that it was both the superior API and slowly but surely winning out over lazy-static.

...but, at the same time, despite the team's best efforts, things like the two deprecated methods on the Error trait do still exist.

I will admit that I haven't noticed much churn in the top half of the list, but those are also niches I don't pay as much attention to.

[Sky2042]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

[ssokolow]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

Ahh, yeah. That sounds/looks right.

I tend to have burntsushi stuff that I'm not already in the habit of using slip my mind because using crates.io is just so much less comfortable than using lib.rs.

[SOF3]

Is there still a real motivation of moving these to std at the cost of future compatibility burden beside concerns over supply chain reliability, which is precisely the point of this RFC?

[jhpratt]

Even if the current crate maintainers were fine with it

I will state for the record that I am not. It is my understanding that, among time, chrono, and jiff, functionality (including via third party extension crates) is substantially similar and that the crate maintainers have acknowledged this. There is no "one size fits all" approach to date-times handling; my recent attempt to create a baseline level of interoperability didn't go very far.

[jhpratt]

And that's not to mention requiring separate reviews for code. If that were required, time would get approximately zero done. As far as I'm aware no one has reviewed the code I write thoroughly since I initially took the crate over in 2019.

[BurntSushi]

I will state for the record that I am not. It is my understanding that, among time, chrono, and jiff, functionality (including via third party extension crates) is substantially similar and that the crate maintainers have acknowledged this.

To be clear, I have acknowledged that this is the case for chrono and time. But not for Jiff. :-) I do not think that jiff is substantially similar to chrono or time in terms of its capabilities. Happy to chat more about that elsewhere though if you want to dig into it.

[jhpratt]

Sure. Admittedly my point was more that there's no clear, unambiguous "winner".

[Lokathor]

Do you have any numbers or other indications to demonstrate that the reliability of software has been hurt by the status quo? I simply don't accept the basic claim that decentralized code is bad and that we can solve things via centralizing. If anything, the rust project is one giant indication that centralization is a quick way to bury a small amount of people under a massive amount of work, until it all slows to a crawl because too many things are looked at by too few eyes.

[JohnDowson]

But is this RFC about centralisation of development, or is it about

1. ) giving users a seal of quality for some third party crates
2. ) providing funding to the maintainers of such approved crates

To me it reads like it's more of the latter.

[Lokathor]

The entire thing is about moving approved crates into an approved github org so that only approved people can access any of it. The goal is a stamp of approval, the mechanism is centralization.

[bdaehlie]

The maintainers of the crates that move into the ESL would continue to be the maintainers. Anyone could contribute to the crates with PRs, same as before.

The GitHub org would enforce some best practices, particularly with regards to security, and additional resources would be available to help with maintenance. I think this is important for the ecosystem's most commonly used crates. Having them differ in terms of compliance with best practices, security policy and resources, and implementation of infrastructure and controls across many different GitHub orgs is not a good thing for Rust security, and it's probably more work in aggregate.

[Lokathor]

So everything is the same, except when it's not the same? That seems incoherent.

I don't ask all this totally out of nothing. tinyvec is a crate that's available in the rust playground. I take this fact as a burden to be extremely careful of. How, concretely, does me moving my repo from my GitHub acocunt to your theoretical org meaningfully change anything for the better? Can you point to actual security situations that have occurred with rust that would have been solved if this RFC had been implemented before the event?

[the8472]

I meant this one: rust-lang/crates.io#11131
It's basically outsourcing the process management of creating a trusted release to github (and other trusted publishing platforms). Then the org can limit publishing to bots instead of humans and the bots will only publish after the appropriate reviews.

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

[epage]

Slippery slopes...

"Trusted Publishing"s name isn't meant to say other forms of publishing aren't trusted but to secure specifically focusing on the CI publishing process. The misleading name already caught on though.

There are trade offs with the current state of bot-based publishing such that we should not require only that. Without requiring bot-based publishing, you then need to allow direct pushes unless you make the release process onerous.

[matthieu-m]

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

I personally would wish for quorums, here.

Automate the publication to crates.io, certainly, but then leave the crate in quarantine mode until other maintainers or dedicated auditors have vetted the published version.

Even small quorums (1 single independent auditor) would massively raise the bar compared to the statu quo.

(On projects where a single maintainer could also trigger a release by a bot, which is likely, you'd want a minimum quorum of 2, obviously)

[the8472]

A) a trusted publishing process may already have a quorum on the CI side (github deployment environments can require multiple approvers), so reimplementing the same functionality on cratesio seems redundant.

B) If we're involving independent authors this seems like out of the hands of the maintainers, in that case "quarantine" would be the wrong framing imo. I think users being able to filter crates on cargo add and cargo update by whatever security policies they want to apply would be more appropriate than making it a global decision.
AIUI big orgs already run their own crate mirrors where they filter which crates can be pulled.

[epage]

Depending on how its done, quarantining packages could cause major problems for workspace maintainers. It would help if we had a whole new publish API but that has been talked about for years. Even once you have that, there can be reason to publish a workspace in fragments.

[clarfonthey]

I don't think that this RFC is a good idea for a couple reasons.

The first is, quite frankly, I don't believe that this project will actually improve the situation, but in fact, make it worse. I do not believe that the Rust Foundation will be able to secure more funding to develop the necessary crates than these crates will ultimately be able to get on their own, which I'll get into later. I also believe that by "spreading thin," changes to these crates will ultimately be reviewed less often and lest thoroughly, meaning that it only increases the risk of "supply chain" issues it purports to prevent.

The second is that, flat out, this does not seem to understand the primary reasons why "supply chain" attacks happen, and thus will fail catastrophically to prevent them. The "secret sauce" involves some combination of the below ingredients:

• Widespread use
• High demand for changes
• A small number of maintainers
• A large amount of work per maintainer

Rust mitigates some of these issues, but not really. While unsafe code is heavily scrutinised, a large portion of the Rust ecosystem still relies heavily on external code without such promises. Even the Rust compiler itself, despite strongly working toward "oxidising" its internal dependencies (Rustup recently moved to being RusTLS only, for example), it still relies on massive unsafe code bases like LLVM and Git. While the number of these will decrease over time, I don't expect this to ever fully go away; even if all of the code Rust itself uses is written in Rust, it can still call down to the OS which may not be. And this doesn't even begin to approach exploits which don't even touch unsafe code.

It also particularly points out the XZ vulnerability, which I think is a perfect example that could happen in the Rust ecosystem right now, given the right circumstances. Right now, we have a small number of maintainers on each team which are expected to sign off on a very large number of changes created by anyone, and the Rust foundation has approximately 12 employees at the time of writing, with fewer people actually managing changes to the code itself.

But don't worry! The power of Rust is that many corporations find it in their best interest to hire people full-time as maintainers too!

Do you trust Microsoft, or Amazon, or Google to always hold the community's best interests at heart and not once, never, ever potentially exploit it? Do you think that this does not put them in an excellent position to leverage their community trust for exploiting any number of things, ever, in perpetuity? Do you trust every person in a position like this to properly back out and tell the project what's going on instead of exploiting their position like their employer wants?

Ultimately, there isn't a great solution for "supply chain" attacks that the tech industry wants, because its misdoings are the reason why such attacks exist in the first place. Moving crates into central control and telling people that the foundation has solved the supply chain problem, however, is actively harmful, because it hasn't solved the problem and can't.

I also would love to see more of these crates become supported and maintained by the entire community, but that's not going to happen by just waving a bureaucratic wand. And I think that pretending that the problem is solved will only make Rust a ripe target for these attacks.

[hawkw]

The drawbacks elaborated in this section are likely to be significant challenges facing such an effort. I feel the current discussion of these difficulties furnished here doesn't sufficiently discuss how these challenges will be handled. In particular, I have to say that simply stating "we will make the difficult decisions" and "we will reach consensus while keeping analysis paralysis at bay" does not inspire confidence in the ESL effort. These are serious issues, and the ways in which they will be addressed is, in my opinion, the most important part of a serious proposal to establish something like the ESL. Similar issues of community conflict and opaque decisionmaking processes have caused substantial problems for the Rust community in the past, and a proposal to establish "blessed" crates that doesn't anticipate such issues and propose a detailed plan for how they will be addressed seems likely to cause more conflict than it resolves.

I think these issues deserve substantially more discussion and forethought than is currently presented in this RFC. They cannot just be handwaved away by saying "we will make the right decisions".

[bdaehlie]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

[Manishearth]

It's worth noting that previous such attempts for a blessed "extended stdlib" died on similar rocks. We have had tremendous conflict in the past when it comes to Rust "picking winners", even with situations where there's just the perception of Rust picking a winner. Folks have quit the community due to this type of thing.

There's some number of libraries where Rust chose to tip the scales (libc, regex, etc) by making them Official Libs Team projects, and that's mostly cases where the ecosystem mostly settled on a very clear "winner" (like regex), or where it's the type of thing that really should be managed by Rust (libc). Arguably, syn could fall in that category.

There's also a worry of interop: even if you bless regex as a winner, codebases using regex can easily interoperate with codebases using other regex engines because it's just a library you call. This is different from, say, picking a serialization library or async runtime, where the decision can be much more viral.

(syn is an interesting example because it's a massive API surface which tends to be a problem for ecosystem interop, but on the other hand 99% of syn users use it as an internal dependency, so it ends up mostly being okay. mostly)

I think it's possible to come up with a set of principles for the types of crates where it is acceptable to pick winners. I think it'll take significant effort, but it's doable. I think that to be a prerequisite for such an RFC; otherwise we risk significant drama and hurt feelings in the future with bad decisions.

At the end of the day, making a decision is important even if it is not "perfect" by a given set of standards.

I strongly disagree with this. It is not important to uplift a library for any given function. It's a nice thing to have. The downsides here are tremendous, and their associated costs are measured in people.

[hawkw]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

If the intent is to use existing decisionmaking processes within the Rust foundation here, I think it would be worthwhile to explicitly state that, at the very least. Ideally there would be some discussion of how these processes apply to the decisions necessary to establish the ESL.

Fundamentally though, I agree strongly with the points in @Manishearth's comment. What you are proposing is something that has the potential to cause a great deal of conflict in the community, and previous efforts to do similar things have failed largely due to that conflict. How are you going to solve those problems this time?

If the goal is to allow users to confidently depend on the crates in the ESL, then it is necessary to establish community confidence in the process through which those crates are selected and through which technical decisions are made in the course of their maintenance. What you want to do here requires getting both maintainers and users of crates on board with the proposal. I think that the right time to sow the seeds of such a sense of confidence is now. Failure to convince the community that the processes used to make these decisions are fair and robust means that the best-case outcome for the ESL is a future where most users prefer to continue depending on independently-maintained crates due to higher levels of confidence in their decision-making processes than those of the crates in the ESL.

[Manishearth]

If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

Those processes have historically been unable to successfully make difficult decisions around elevating ecosystem projects. Really, anything involving people and feelings over code is super tricky and is not what the Rust decisionmaking process is designed for.

It's understandably not obvious from the history, but I'd say figuring this out is the core prerequisite for any such proposal.

[hawkw]

For the record, I want to make it clear that I am not expressing opposition to the idea of having something like the ESL. Instead, I'm pointing out that there is a reason that there are so many previous attempts to do something like this — if it had worked in the past, we wouldn't have to be proposing it now. If this attempt is to succeed, it must address the issues that have led to the failures of past attempts. Perhaps it will do so successfully, but the text of this RFC doesn't give me a reason to believe that.

[the8472]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

[epage]

Async, cli parsers, and many other cases involve trade offs and selecting which trade offs to bless will be difficult.

Some ideas

• Bless more than one, like blessed.rs sometimes does
• Bless the most minimal one (std-like)
• Bless the most flexible (full batteries)

[ssokolow]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

Honestly, I think it's premature to bless any async runtime before the "tell a complete async story" process is far enough along to start abstracting the APIs that tie projects to specific runtimes and adding those to the standard library as traits.

It wasn't that long ago that we got the "async fn in trait" MVP.

[the8472]

We have git as a tool to share code. We have github as a community platform to make the code-sharing and communication around it public.
We have cargo vet (and crev) as tools. What we're lacking is a visible platform for aggregating reviews, the social dynamics around 3rd-party code reviews. It's not a publishable good.

So I agree it doesn't go far enough. But I think rather than centralizing a few critical pieces a more scalable solution would be to record the figurative eyeballs that went over all the code, and the lack of eyeballs too.

Netflix, Huawei and the sovereign tech fund assigned engineers to review this code is useful information.
10000 people use this library but nobody ever reviewed it is useful information too.

If everyone (and every company) reviews in silence we repeat efforts and don't know who else already did the thing. It feels very much like a coordination problem. Nobody reviews a thing because reviewing everything is too much work.

This could start with surfacing such information on crates.io.
And "review crate of the day" kinds of efforts, make it a vscode plugin, and...

Otherwise I think the issue will be that any hypothetical supply-chain attacker will move their focus to "widespread use, but just outside the ESL scope" crates, which will remain unreviewed...

[epage]

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

There is that and other tacticts which we should work towards to improve supply chain management. I worry this might distract us from that larger problem because this can't cover everyone's needs but people will likely be optimistic about this effort

[Manishearth]

What we're lacking is a visible platform for aggregating reviews

n.b. libs.rs does show vet. as does deps.dev.

My comment below suggests a way we could make great use of cargo vet for attaining the goals in this RFC.

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

Yes, 100%.

[Manishearth]

I think this might be selling cargo vet short. The design is not for one to trust random audits; it was thus designed precisely because trust is multidimensional and contextual. It is a framework upon which to build a trust model. cargo-vet doesn't care about a project's security policies because cargo vet shouldn't be making that decision: instead, it lets one define their own criteria for what constitutes "good security policies" and filter based on that.

A very feasible way of achieving many (not all) of the goals of this RFC would be to define a set of criteria, and have some blessed audit repository for those criteria¹. It could be automated such that crates whose maintainers are trusted to follow these criteria automatically get audits published alongside their crate publishes. Sites like lib.rs could be petitioned to elevate such audits with a badge.

"Trust the entity that produced the code in the first place" is definitely a possible thing to represent within cargo-vet's framework.

(Potentially crates.io as well, though that's the threshold where you would need an RFC, up to this point this can be done entirely as a community effort).

Footnotes

1. audits retain their source: it is possible to distinguish a Google audit from a Mozilla audit from a rustlang audit. ↩

[epage]

This should be out of scope. Renames are a breaking change and requiring one to add/remove branding is a bad idea.

[Manishearth]

In theory cargo/crates.io could support "redirect" renames. It's a thing that was briefly discussed during the namespacing RFC, it didn't end up happening, but it's ... something that could be proposed.

I agree that without such a change it's probably not going to work.

[epage]

We should not count on being able to do so though as I expect some tricky problems in the resolver and Cargo's ideals around ‘Cargo.lock` files.

[epage]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

[epage]

(I have similar concerns about several others of these ideals)

[the8472]

I would even go a bit further, I think calling these requirements to be trustworthy is corporate-biased and mildly offensive to individual maintainers.
Saying that you're not trustworthy because you can't guarantee to maintain something well into the future or because you don't want to deal with the security stress and just want soundness bugs treated like any other bug doesn't make a dependency un-trustworthy. It just might make it fall below the level of unpaid support that some corporations might want.

But well

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

To me trustworthiness in something that's provided for free only means non-malicious and not having a track-record of bad decisions. Everything else sounds more like paid support or nice-to-haves.

Some things don't require eternal vigilance, e.g. hex hasn't seen a release in 4 years and imo this doesn't impinge on its trustworthiness, it might eventually become a problem for its usefulness.

[jieyouxu]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

To add to this, even the compiler, standard library, and various rust-lang/* crates like cc or libc have limited reviewer bandwidth, especially in specialized areas of the codebases.

• Has sufficient maintainer bandwidth to publish security updates in a reasonable timeframe
• Has reasonable code review and testing standards

Are really, really hard problems. I fully echo the

assuming more than can be expected or will give people a false sense of expectations for the result

concern.

[jswrenn]

How would removal work in practice? If the name of the crate is what conveys its membership in the ESL, what happens when that crate is removed from the ESL? Are its versions yanked?

[jswrenn]

How will the ESL ensure that existing maintainers are non-malicious? How will the ESL ensure that new maintainers of ESL crates are non-malicious?

[frewsxcv]

And similarly, who decides whom is qualified?

[BurntSushi]

I feel like this is largely a non-issue. Who decides who is on libs? Or the lang team? How do we know everyone on the teams is not malicious? How do we know they are qualified?

[frewsxcv]

If it's not important and up for debate, then what is this line for?

[BurntSushi]

It seems like it's elaborating on an important goal to me.

[epage]

This is a real tension where I have to decide between

• Being blessed or fading into obscurity
• The policies if they are too heavy

[jswrenn]

The drawback of picking winners isn't that folks "could be upset", it's the (good) reasons for which they would be made upset. If you maintain a crate whose competitor is selected for inclusion in the ESL, there will begin to be tremendous pressure to avoid your crate. If you have an idea for an alternative approach to a problem "solved" by the ESL, there will be tremendous pressure to avoid your crate. The ESL has the potential to stifle innovation in the crate ecosystem.

Had the ESL existed in 2018, I don't think I'd be paying my bills by maintaining one of the crates I maintain today; there simply would have been too much pressure to use the then-dominant alternative.

[BurntSushi]

I somewhat feel like this is already true today. I think the fundamental difference between the status quo and this proposal is that this proposal has a (presumably small) group of people making a determination about the winner as compared today where a winner tends to manifest via inertia or other pressures. But we certainly don't need an ESL to have the problem of it being almost impossible to unseat an existing dominant solution.

[Lokathor]

But I think we can definitely avoid making it worse at the least

[hawkw]

There's also the issue that, in many cases, there may be multiple crates for a given purpose which make very different design decisions. Often, these design decisions are equally "correct", and the right choice depends on the particular use case. In a situation like this where two or more crates that have a comparable level of maintenance quality make substantially different and incompatible design choices, what would the ESL do? Would both alternatives be added to the ESL? Or would the ESL just choose one such crate and leave users whose particular use case prefers the other design without an ESL-blessed option?

[fintelia]

Also worth calling out that choosing only one competitor means that the other ESL crates are forced to make the same pick. Basically...

We'd love for your crate to be part of ESL, but of course you'd have to switch from fastrand to rand. I'm sure your users won't mind!

[hawkw]

Here's a fun question that it would be nice to have an answer to: what happens if an ESL crate depends on a non-ESL crate? Now, you have a crate that's part of the trusted base of vetted libraries which, nonetheless, has the potential to "get xz'd" by its non-ESL dependency. Would ESL crates be forbidden from depending on non-ESL code? Would adopting a crate into the ESL require ESLing its entire dependency tree?

[epage]

I worry this will be too restrictive. I'm splitting up toml. If it was in the ESL, I'd have to get those lower level crates to be blessed to a similar level which also unduely raises their visibility withinea blessed list.

I worry this can also be stagnating with other dependencies.

[the8472]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.
https://forge.rust-lang.org/policies/crate-ownership.html

[epage]

Another example: I'm working on a replacement for libtest. It has its own cli parser and soon a json writer to meet very specific design requirements. Abandoning those requirements would be rough blessing each part would also.

In both examples, some of the packages are also being developed on an experimental basis. And the libtest project just left rust-lang org because of how strict the requirements are, even for experiments.

[Lokathor]

It's unfortunately demonstrably true that, until we have multi-threaded rustc, it's faster for builds to split crates into multiple sub-crates as often as possible. We probably want to encourage large crates used by many people to break up their code as much as possible, rather than encouraging single large crates.

[epage]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.

The Project has that but this is separate. That policy is also very strict in that something can only be Intentionally provided by the Project or its solely Internal to the project. Some of the cases I'm referring to would be Intention for the ESL or non-blessed within the ESL but still Intentionally for end-users.

[epage]

Have you talked with them for what lessons they have just doing a fraction of what this RFC does?

[oxalica]

Is there a possibility of blessing blessed.rs itself, making it into a semi-or-fully-official list? I think it's another alternative solution worth mentioning.

This would avoid the cost of maintaining crates themselves, bless multiple crates for different design goals. Newborn crates competing existing crates (with new design, new goals, new algorithm or etc) can also be less painful (as mentioned by @jswrenn https://github.com/rust-lang/rfcs/pull/3810/files#r2080592616) and they can be added to (not replace!) the list later if some conditions are met.

[epage]

As noted in https://github.com/rust-lang/rfcs/pull/3810/files#r2080548607 this runs counter to that RFC as that is about a cohesive API while this is about an organization.

[epage]

What lessons can we learn from these prior art?

[erickt]

Should this RFC discuss if it will be possible to remove crates from the ESL, or would we allow for duplicate functionally? I recall stdx was hoping to support removing crates, but I imagine it could be very disruptive to the community.

[epage]

This would be longer support than we off Rust itself

(yes, this is an example)

[comex]

Not really. The text is talking about how long to support an old SemVer-major version after releasing a new one. Rust itself never has SemVer-major updates, so the issue never arises. Theoretically at least.

[epage]

While this is an example, it highlights the assumed maturity for ESL.

Is there a place for younger crates that fill a core need within ESL? I'm again thinking of my libtest work once its further along.

[ssokolow]

...and what role does rust-lang-nursery play in this vision? I'm not sure it's even easy for people to learn what membership in it means now.

[epage]

Not to debate policy but just so I don't forget, there might want to be policies on this. In most cases, there is little difference between minor and update. When supporting for 2 years, there are major differences

• The more majors, the more work to backport, patch, and release
• Without majors, you can't patch. MSRV bumps and "risky" changes (major refactors) should necesisitate a minor

[Lokathor]

I think we should not accept pre-1.0 crates into the ESL. I'm not even joking.

[riking]

Being on a 0.x.x version hampers so much of your ability to perform semver updates; imposing a technical requirement for a non-0.x version does seem justified.

[epage]

Not to debate policy but just so I don't forget, consider the MSRV applying to the package and not every version.

What I mean is that a previous but supported minor satisfying the MSRV should count, even if the latest version doesn't. Maintainers will have flexibility to bump MSRV while having an incentive to not do it too often because it means more versions to support, creating more work, see https://github.com/rust-lang/rfcs/pull/3810/files#r2080681065

[epage]

Similarly about not wanting to forget, consider whether "N versions backs" actually meets people's needs. That means the MSRV can change every 6 weeks. Those with a direct MSRV requirement usually need predictable, stable stuff. Calendar based approaches might work better.

[Lokathor]

If were legislating MSRV we'd need to actually pick a policy on cargo features affecting MSRV, which means the cargo resolver should support that logic as well. Not that we can't do any of that, but it's complicated

[epage]

From where I am currently out, I see per-feature MSRVs to be highly unlikely.

[Lokathor]

If that can't be supported in some way, then "two years of msrv" is effectively a joke proposal.

[Lokathor]

epage was just taking notes on what the ESL org would have to cover eventually, I'm mostly just adding to the notes pile.

and i know that RFCs aren't always implemented exactly as first written, but if there's not really a remotely realistic way to implement what the RFC says, then the RFC needs to say something else. The current RFC text currently states a pretty specific policy.

[BurntSushi]

Okay.

[epage]

Along the lines of MSRVs is the conflict between long MSRVs and build times which is also another strong care about. I was analyzing someones dependencies recently and there were a lot of deps that exist solely for MSRV. Some could be handled through polyfills while others will require cfg(version) / cfg(accessible) to be stabilized. This is another reason to consider MSRV to be about the project as a whole and not apply to every major or minor version.

[nicoburns]

no_std support is another major source of otherwise unnecessary dependencies. Many projects unconditionally depend on libraries like libm and hashbrown because there is no ergonomic way to specify dependencies that are only enabled when not(feature = "std"). So you either have to add something like a no_std or libm feature (which is inconvenient for consumers of your crate) or unconditionally depend on the necessary crates.

[epage]

I wonder what all cases there are like this and if there are some high leverage changes that could be made to avoid this, like getting HashMap into alloc with a no-std hasher available.

[epage]

Not to debate policy but just so I don't forget, we should consider a little more nuance here.

Problems with releasing new major versions for why they should be throttled / avoided

• Work to upgrade
• Interop when in a public API of another package
• Compile times from multiple copies

Not everything is big and not everything is meant to be in a public API. As for being disruptive, we should probably encourage (but not require) breaking releases to be removal of deprecated functionality as discussed on matklad's blog and as I try to do with clap and winnow as much as possible.

[Lokathor]

I don't think I understand what you're trying to get at here. When you say "major" do you mean the first number in the three number sequence? cargo's vision of versions is to treat the first non-zero version as the "breaking change" number, and the next numbers as compatible, right? So how is 0.x.y somehow better than x.y.z? An incompatible version is just incompatible.

[epage]

The topic I'm replying to and my comment and "major" to refer to the relative position in the version number and not the absolute position (wish we had established names for these)

My comment is speaking to how to manage breaking releases.

[epage]

I think I saw where some of the confusion came from and made an edit that I hope helps.

[epage]

What about non-malicious but unfavored changes?

This can range from

• A maintainer making the API more non-ergonic for purity purposes (highly debateable on whether this is a problem)
• A maintainer shipping pre-built binaries (we saw how that went)

Flipping the perspective on this, I know with clap I've appreciated having WG-CLI as a sounding board for things I thought might be a good idea but I was unsure how they would be received.

[ssokolow]

*nod* APIs becoming less ergonomic is definitely a concern.

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths (the only reason I choose it over gumdrop's offering of a derive API with faster build times as I do in cases where I can just say "non-UTF-8 paths not supported" like actix-web www roots and config file paths) so opaque last time I looked at it that I've built a habit of just rg-ing my way to one of my existing projects which contains a suitable OsString-processing pipeline and copy-pasting the relevant snippet to the new project without even trying to remember why it works anymore.

Clap 3's "vendored StructOpt" API? Comprehensible. Clap 4's overindulgence in rustdoc-hostile, rustc-error-message-hostile type-level dispatch obfuscated by weaknesses in the concept of derive macros? Not really.

For me, clap's API has become a bag of incantations with my prime use-case for it achieved through a piece of tribal knowledge that you blessed me with in some comment on the issue tracker that I forgot to bookmark... and I'm one of those people who makes an active effort to understand APIs before I use them and to maintain that understanding. (eg. I "make things harder than necessary" when implementing custom Serde work by trying to save the brute-force "just impl a whole new Serialize" option as a last resort until I've proven to myself that there's no higher-level way you're meant to be able to do something.)

[epage]

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths

While a bit off topic for this RFC, OsString paths "just work" which was a large reason for the clap v4 design as before where you had to know they weren't supported natively and find the right incantation to get them to work.

[ssokolow]

They "just work(ed)" if you "just want an OsString". Last time I wasn't just copy-pasting a previous successful use, they were a frustrating and under-documented type-system puzzle-box if you wanted to plumb in some custom validation and end up with a PathBuf field on your struct. (But, as soon as you told me the magic incantation, they did indeed "just work".)

With StructOpt and Clap v3, I had no trouble understanding from the docs how to specify that I wanted the argument ingested as an OsString, passed to a validator function, and then ending up in a PathBuf... even if the APIs were suboptimal.

[oxalica]

What about non-malicious but unfavored changes?

There is already an example in the ecosystem,though not quite in this RFC's scope, gluon-lang/lsp-types#284. Some downstream users are so unfavor the changes and start creating forks. I think it's a lose-lose for the ecosystem. The problem will be worse if the crate in question is in ESL.

[Kijewski]

Maybe you could indent the content under a bullet point to make it more readble?

1. **Title**

   Description

   More description

2. **Next point**

   It has a description, too.

1. Title

   Description

   More description

2. Next point

   It has a description, too.

[nicoburns]

I'm glad to see someone trying to address the problem of supply chain attacks. I think it's an important issue which is currently very difficult to mitigate against.

However I do not think that the presented solution of centralising maintenance of key crates is a good one. I agree with @clarfonthey that the constraint which creates the opportunity for supply chain attacks is lack of available maintainer resources, and that this proposal is likely to reduce the availability of such resources not increase them.

I think the philosophy behind Cargo Vet where vetting/auditing is decoupled from maintaining/owning is the right one, and if there is funding available, I would love to see this model expanded via things like:

• An official, well-maintained version of diff.rs. This website allows people to 1. easily read the code actually published to crates.io (not just the code checked into git which may be entirely different) and 2. see a diff between versions of that published code. To me, these are foundational capabilities for enabling audits.
• Vetting/auditing functionality built into this "diff.rs" UI (e.g. something similar to Github review UI) such that people can easily create audits/reviews via the UI. These would then be saved to a central database.
• Metadata attached to audits with things like standardised categories which denote the depth of review that has been undertaken and the reviewer's confidence level in their review.
• Tooling built-in to cargo which allows users to specify criteria such an "N reviewers (with an optional allowlist of specific reviewers or organisations) must have approved" and/or "there must be less than N negative audits" (perhaps with an additional "warn" outcome that prompts the user to go and read the audits and/or audit the code themselves).
• Pushing crate authors towards MFA
• Expanded support for (emphasis on) signed crate releases (e.g. support for allowlisting certain crates only if new releases are signed by a trusted signing key and prompting for manual verification if a new key is used)
• Directly funding trusted people to spend their time producing audits

Trusting specific crates/publishers is all well and good, but the supply chain attacks I would be most concerned about are when crate maintenance changes hands or a users credentials are compromised. To mitigate against these kind of attacks one really needs each published version to be verified, not just the crate or it's publishers. Given how many users of the really key crates there are, it should be easy to get enough reviews if making one is easy enough.

I think lists of approved crates/authors/auditors/organisations would be useful. But I see no reason why there needs to be a single list. We could perhaps have an "official" rust foundation list that is quite conservative. But also allow other organisations to publish lists which people might decide to either trust wholesale and pick and choose pieces of to form their own lists (e.g. corporations like Mozilla, Google, Amazon, Dropbox, etc; open source collectives like Servo, Linebender, Bevy, etc; and even individuals that may be trusted by the community).

[joshka]

One thing that struck me when seeing the ESL acronym is that it ESL often stands for "English as a Second Language" in a large number of English speaking countries. This may not be the sort of acronym overload that you want for a project like this. Either Stdx or StdEx (more aligned with the naming of rust extension traits) both seem like like much better names for this.

I've intentionally added this comment here near the end of the document rather than at the top of the page where ESL is first introduced. Hopefully this will avoid too much of a bikeshed discussion happening and distracting from the much more substantive conversations.

[ssokolow]

We don't want a std 2.0 (cf. python 2/3 breakage)

Not to mention that in C++, stability requirements (in that case, ABI stability) led to std::regex being slower than shelling out to PHP.
We already have std::sync::mpsc in std. Do I ever use it? No, I used to use crossbeam-channel for the MPMC API that it still doesn't provide, despite now wrapping a vendored copy of that. Now? I use Flume because it also does async.

There is always the option of not repeating the mistakes that languages have made in the past. I'm not trying to be trite but to me these aren't necessarily arguments against expanding or changing the std library,

The problem is that clairvoyance is an unsolved problem. Being extremely conservative about how much stuff is baked into the library you've committed to never having a "Python2 to Python3" transition on is "not repeating the mistakes that languages have made in the past".

these are arguments against expanding or changing the std library without long term resource commitments and a well known plan. I would also caution people against conflating resource constraints with technical constraints. I do think whatever your thoughts on Go as a language, I think the Go team has conclusively proven that given enough resources and direction these are very surmountable challenges by just investing in quality maintenance work.

Personally, I think at least half of Go's solution to it being an insurmountable problem is WONTFIX NOTABUG-ing the problem out of existence. It's much easier to design an API which lasts if you just define the unsolved parts as out-of-scope.

Go is what you get when people think all C needed is garbage-collection, a perforated implementation of memory safety, a concurrency model that was all the rage in the 1990s, and a strong anti-FFI belief that "cgo is not Go" because things like extending Go-style easy cross-compilation to FFI is hard work that requires cooperation with people like the LLVM or GCC developers.

But that obviously doesn't actually address these concerns for us as the Rust Foundation has significantly fewer resources than Google. To me this is an even stronger argument for using inclusion to the std library as a litmus test. If the rust foundation doesn't want to commit resources to including something in the std library (or alternatively, they don't believe such a thing belongs there) then why are we committing resources to an ESL for that same code? If the answer is just security it's my personal opinion that sponsoring regular security audits would be infinitely more cost effective. Otherwise the ESL will incur large costs with very little benefit.

Please correct me you disagree here but again I just don't see a way these ESL libraries can achieve this middle ground where they are stable, trusted, reliable, and secure but still able to be nimble, make breaking changes, and iterate fast without dedicating roughly the amount of resources required to include that same code into the std library.

I can agree with that. Every time some kind of stdx has come up, my perspective, has always been "OK... but where is this extra manpower gonna come from?"

Edit: In case it's not clear, my general thrust here isn't that these things should be in the std library, just that long term blessing this code has roughly the same resource commitments as including them in the std library, and if we don't want to do that then we shouldn't include them in an ESL either.

I disagree that they should be in the standard library for the same reason that, when you look at things like accessing members of collections, C++ reveals itself to be three different languages in a trench coat, refusing to admit that C++ failed and C++ v2.0 failed by disguising the fact that they underwent a Python2 vs. Python3 vs. Python4 transition as, instead, a "Python 2.x has urllib and urllib2 in the standard library" situation.

[Diggsey]

I think there's a nugget of value in this RFC: specifically a set of "higher standards" that crate authors can choose to follow, where those standards are agreed upon in a centralized way.

These parts on the other hand seem actively harmful:

• Moving crates to a central github org is pointless and creates a single point of failure.
• Naming crates after ESL creates harmful churn for users of the crates, and the idea that crates will be created to be ESL crates from the start is pure hubris, and will lead to a ton of garbage ESL crates.

I think what could work would be a decentralised trust model:

• Any organisation can announce on a crate's crates.io page "I trust this crate revision". The amount of scrutiny in this review would be entirely up to the organisation.
• As a user, I can use a cargo audit like tool to find dependencies that are not trusted by any of a set of reviewers that I trust.

With this approach, any person or organisation can create a "brand" around their crate reviews.

Optionally, a Rust ESL team could maintain a list of "trusted reviewers". They would set the criteria for what warrants inclusion in this list, but might include:

• Having a certain number of reviews.
• Being independent of other reviewers in the list.
• Having certain standards of review.
• etc.

Following from that, any crate revision which has been given the seal of approval by at least two or three different reviewers on this list would be eligable to be part of the ESL.

Notably in this setup, the ESL team is concerned only with policy making and maintaining this reviewers list, and is freed from the more time consuming work of reviewing the many crates involved. Furthermore, it gives large companies which might already be doing some level of review of their dependencies the ability to feed back that value into the ecosystem, without ESL crates being exclusively corporate focused.

[mcobzarenco]

I see a philosophical problem with blessing a subset of dependencies for writing backend code. Rust will suddenly look a lot less like an all-encompassing replacement for C/C++ and instead more like a supped up Go for writing APIs. I don't want to see this.

Just to pick a random one, why have base64 in std? I have been writing Rust since 2013 (since 2016 professionally), and I haven't serialized/deserialized something as base64 once for work.

To my machine learning/scientific computing eye, that's something a backend server cares about. Not to mention, there are multiple base64 libraries with different trade-offs.

[arielb1]

I see a philosophical problem with blessing a subset of dependencies for writing backend code. Rust will suddenly look a lot less like an all-encompassing replacement for C/C++ and instead more like a supped up Go for writing APIs. I don't want to see this.

I think that much of it is that backend servers are where the "Underwriters Laboratories" stamp is most important. For other sorts of programs there are generally other methods that prevent them from burning down your company (people might also want a "safe for embedded" stamp, but that stamp would have different regulatory characteristics than a "safe for backend" stamp).

But I agree that the existence of multiple versions of base64 decoders (my fairly simple Cargo.lock has 3, 4 if you count different versions of base64) is a pretty good argument against having a decoder in libstd, but also a pretty good argument for having a way to "bless" them all.

[VorpalBlade]

I posted this originally over on the reddit thread but I think I should cross-post it here so it comes to the attention of those working on this RFC:

I have a number of concerns with this RFC:

1. It will entrench a number of "local optima" crates even further. For example, serde has flaws: it cannot support true zero copy deserialisation (e.g. mmap file and be able to use it directly as is) without breaking changes that are unlikely to happen. And due to monomorphisation it can bloat the code size and build times (problem for embedded and for projects where build time matter more). The crate Facet by fasterthanlime seems like a really interesting alternative approach that could solve this, but it is in fairly early development still. And don't get me started on tokio, io-uring and non-server async. The non-duplicative clause is especially bad here as it stifles innovation.
2. Just because it is officially blessed won't mean it actually gets maintained. Python had issues with this, where they had to remove standard library modules that went unmaintained (smtp support and some now obscure file format if I remember correctly for example). And in some cases better alternatives came out in the ecosystem and people stopped using the standard library modules (and for those not in the know, they are now using worse modules). In python you shpuld use regex instead of re, requests instead of http.client, pytest instead of unittest. There is a saying in the Python community that "the standard library is where libraries go to die". I fear this would happen here to.
3. Syn which is mentioned in the RFC is slow to compile, surely we can do better, such as unsynn? This is related to the point here.
4. Which hex or base64 crate? The one that compiles to small code (yay, good for embedded and wasm) or the one with SIMD support (good for my 128 core server load where only speed matters).
5. Also, Rustc itself and std depends on multiple crates from the ecosystem. And while they do check those, it is not a like they do a full audit of every dependency, and especially not on every update. How would that interact with this ESL idea? Would rustc only be allowed to use dependencies from ESL?

[arielb1]

I agree. I expect the Rust project to have negative alpha when it comes to finding "the best crate for usecase X". Also, I believe that "find the best crate for usecase X" is best done by an algorithm - such as lib.rs, or a search engine/LLM. It might make sense for rust-lang.org to link to one.

Which does not detract from the Rust community benefiting from a way of knowing that a crate is "reliable", even if not necessarily "the right tool for a particular job", along with a path for crate authors to be able to get their crates to reliable (of course, the Rust Project considering a crate to be "reliable" does not mean that it is owned by the Rust Project. For example, I am quite sure that the Rust Project regards Tokio as a highly reliable crate, but Tokio is run by a separate project).

[abgros]

Will the extended standard library be automatically downloaded by Rustup?

[arielb1]

Moving crates to a central github org is pointless and creates a single point of failure.

Giving maintainers of popular crates the option to put their code in a Github repository with certifiably high security standards and coupling the code on git to the code on crates.io somehow (while letting the maintainer have complete control over the code except in "actual security breach" situations. I mean, Github owns the git repository and the Rust Foundation owns the crates.io index even today and they might act even today in the case of an actual security breach) rather than having maintainers needing to figure out by themselves how to secure github.com/jsmith123/my-crate, sounds to me like a good enough idea if it can be done in a reasonable way.

Of course, if maintainers already have some sort of git repository that they know how to secure, they might not need this help, and the Rust Project should not have a "let us own your repo OR ELSE you won't get a pretty badge" system.

[joshka]

Giving maintainers of popular crates the option to put their code in a Github repository with certifiably high security standards ...

The core benefit of that is that certification is measurable. It would be significantly more useful to be able to verify that those standards are met. Putting the source into a GitHub organization is not the only way to make that happen.

[Diggsey]

Giving maintainers of popular crates the option

Yeah I'm not opposed to giving the option, as long as it's not a requirement.

[burdges]

Non-malicious is subjective: You can be upstanding and well intentioned, but your employer maybe considered inherently malicious, ala government contractors, blockchain companies, advertising companies. It's likely that some nationalities get treated with suspicion by specific governments too, maybe requiring more ongoing auditing or whatever. It's likely better if these problems get addressed behind closed doors by the worried parties, nd the wider community mostly ignores them.

What might make more sense is if crates put "Integration notes" into their repos an/dor issues on what they've done, or plan to do, to worke better with other crates, or what they wish others would do.

[burdges]

I suppose "non-malicious" could be replaced by simply some disclosure policy: The absolute minimum would be that crates should disclose if their maintainers worked for millitary, intelligence, or law enforcement contractors, since those are the parties most likely to exploit an undisclosed exploit.

Auditor reports would be another standard here, but that gets expensive.

[teor2345]

There needs to be a (more) objective criteria here. Otherwise, people, topics, or groups can simply be declared malicious, even though a dependency on the crates they develop is unlikely to intentionally cause security issues.

This is particularly important for a project that seeks to be the One True Source for the One True Crate for X. You don't want to get caught up in arguments about the value or purpose of crates. Those issues are important to discuss and act on, but they need to be out of scope for this process.

(There are already contentious examples in this RFC discussion.)

[burdges]

As a rule, rust code outside the standard library winds up more "opinionated" than the standard library, like even the getrandom crate in rand, and/or more more featurefull. In particular, the standard library tends more towards a simple baseline targeted at most devs, but it absolutely doesn't care if your niche use case require that you fork one of the stc functions.

Those are natural explorations, but you really do notice a big stylaistic difference between std and other crates.

[TimTheBig]

Suggested change

	`itertools` equivalent

[jswrenn]

[itertools maintainer hat on] We make breaking changes much more frequently than the guidelines of this RFC suggest is appropriate for an ESL crate. Our types don't typically appear in public APIs, so this isn't typically problematic for our users. Itertools has long been informally referred to as part of Rust's extended standard library (and gadgets from itertools are regularly uplifted into the standard library), but it doesn't meet the stability and support criteria outlined by this RFC.

[TimTheBig]

Good point, maybe a stable subset of itertools could be part of the ESL

[jswrenn]

Maybe, but if such a subset exists, then why not make it part of the standard library? Again, elements of itertools are regularly uplifted into the standard library.

[ssokolow]

Good point, maybe a stable subset of itertools could be part of the ESL

Wasn't the point that "the stable subset of itertools is 'and gadgets from itertools are regularly uplifted into the standard library'"?

...as in depending on itertools is an alternative take on cargo +nightly build and things being in it instead of std is an indication that more work is needed before they'll be ready to be that API stable?

Premature stabilization can't be forced. Trying just results in "Python 2.x stdlib has urllib and urllib2 and everyone tells you to ignore them in favour of Requests, which contains a urllib3 that will never become part of stdlib". Induction into a "standard library" that results in successful stabilization is an acknowledgement that stabilization has happened organically, paired with a little pruning away of vestigial organs.

[joshka]

This is a perfect example of why focusing on standards may be (at least in some situations) a better idea than focusing on a standard library.

[TimTheBig]

Fair, it would be great to have stuff like ExactSizeIter in std it would be great for performance. That's why ESL as a step between crates and std. Some kind of robust process to uplift things from ESL to std would be great.

[SOF3]

Would we have less bikeshedding over preferences if we call this "officially endorsed libraries" instead of "extended standard library"? The term ESL gives the following (possibly untruthful) impressions:

• that ESL will be shipped with rustup
• that ESL correlates with msrv
• that ESL is maintained by T-libs or equivalent
• that ESL is an official recommendation over non-ESL libraries
• that the barrier for a v2 in an ESL library is very unlikely

[programmerjake]

Would we have less bikeshedding over preferences if we call this "officially endorsed libraries" instead of "extended standard library"?

• that ESL is an official recommendation over non-ESL libraries

for me, "officially endorsed libraries" gives that impression too, so if you're trying to avoid it, you'll want a different term.

[programmerjake]

maybe "well-known libraries"? that doesn't imply Rust is officially maintaining them, but that Rust is recognizing them as a good option, if they suit your purposes.

or something like "officially verified libraries", kinda inspired by what the blue check was on twitter before musk bought it.

[Lokathor]

If it's just about being well known, then there's already ways to search that information up.

[programmerjake]

If it's just about being well known, then there's already ways to search that information up.

maybe "officially well-known libraries", since I think the idea is that Rust is endorsing that the library is fine to use in your dependencies, but not trying to imply that Rust itself is maintaining it or that you must use it over any other libraries if it doesn't quite do what you want.

[tgross35]

(nit) The use cases for quote and cfg-if are likely to be covered by std at some point in the hopefully-not-too-distant future: rust-lang/rust#115585 rust-lang/rust#54722

[TimTheBig]

I think encoding/file formats should not be included as it's very specific to a use case, with the exception of those needed for http.

[Lokathor]

http is as specific a use case as "open a png file". if you're going to do one esl thing there's no reason not to do both esl things.

not that i think this esl thing is a good idea

[TimTheBig]

I feel that this should be part of lib proc_macro, I have never seen a proc_macro written without it or a library that depends on it. This the most downloaded crate with over 870,000,000 downloads.

[programmerjake]

I disagree that syn should be part of proc_macro specifically because rust's syntax is still changing and you want to be able to have updated AST structures that support the new syntax. this is why there's syn 2.0, and i expect there will be a 3.0 in the next few years or sooner.

[TimTheBig]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

[ssokolow]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

I could get behind giving https://github.com/rust-secure-code/safety-dance a friend.

[TimTheBig]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

I could get behind giving https://github.com/rust-secure-code/safety-dance a friend.

Auditing crates and making some kind of standard process that any can use before a release would be a lot cheaper too.

[teor2345]

There should be an exception here for platforms where the functionality isn't needed or can't be implemented. We might also want exceptions where it would take a large amount of (unfunded) effort to make a crate work.

How would the relationship between Rust tier approvals and the ESL work?
Would approvals be gated on the ESL, or would the ESL have to adapt to new approvals?

[teor2345]

I don't understand why choosing a single crate for each purpose is important here. Or maybe I've misunderstood the RFC, and "which one should be preferred" is much more granular, and "not duplicating" is narrowly defined.

For example, could there be multiple async executors (for different use cases), such as performance, embedded, and ergonomics?
Or multiple CLI parsers with different feature sets?

[teor2345]

I don't see a clear rationale for combining the three motivations of this RFC into a single process/organisation. Either way, it's important to mention the alternative: tackling them each separately.

It feels like this rationale is mainly about motivation 2: the blessing of the One True Dependency for each purpose.
What if that was a separate, narrowly-scoped RFC?
Would we come up with a different set of tradeoffs?
Would there be much connection to ecosystem security at all? Or would it mainly be about discoverability?

Similarly, we could treat motivation 1 as a separate task, codify a set of security policies, assess crates, and list/badge those that pass. This would have a longer-term impact on ecosystem security.

And we could provide security support to all popular crates, regardless of affiliation (motivation 3). This would have the greatest immediate impact on the security of the ecosystem. Pay people to audit crates, and provide security fixes. Pay them to mentor interested security volunteers.

Given the existing discussions on this RFC, it seems like splitting it would be more likely to achieve (some of) its goals.

[fintelia]

I don't think the RFC ever actually justifies why there can't be duplicate functionality between ESL crates. The closest is pointing out that having a single endorsed crate making it easier for users to pick. But that doesn't seem like a sufficient argument to justify the significant downsides (discussed elsewhere in the thread). It wouldn't completely eliminate the concerns around "picking winners" but not having a fixed quota would at least slightly mitigate them.

Really it feels like there's two pieces of this proposal. One is a set of suggestions around funding core ecosystem crates, trying to align support expectations (MSRV, etc.), and combining effort spent on infrastructure. That all seems completely virtuous and worthwhile to me. But there's also another piece that an uncharitable reader could interpret as trying to co-opt the crates.io ecosystem: Picking a single winner for each category, coercing maintainers to join with an implied threat of forking their crates, and the underlying goal of sharply decreasing the usage of non-ESL crates