[WIP] Forbid object lifetime changing pointer casts

[BoxyUwU]

Fixes #136702

r? @ghost

[BoxyUwU]

@bors try

[bors]

⌛ Trying commit d5ebeac with merge 44f3504...

[bors]

☀️ Try build successful - checks-actions
Build commit: 44f3504 (44f3504e96c944ae54fc72b5f5008f53f7eda001)

[BoxyUwU]

@craterbot check

[craterbot]

👌 Experiment pr-136776 created and queued.
🤖 Automatically detected try build 44f3504
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🚧 Experiment pr-136776 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-136776 is completed!
📊 169 regressed and 4 fixed (580506 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[RalfJung]

Most of these are on github; in terms of crates.io regressions all we have is:

• may
• a bunch of crates using metrics, see e.g. this (for metrics-0.23) or this (for metrics-0.24, the latest version)

Overall, 142 regressions are caused by metrics and 14 by may; if we ca get fixed versions of those crates out that seems to mostly cover it.

EDIT: Ah, there's also cogo.

[traviscross]

We discussed this in the lang triage call today. We wanted to think more about it, so we're leaving it nominated to discuss again.

[tmandry]

@BoxyUwU Do you think it would be possible to implement this as an FCW? We talked about this in lang triage today and would prefer to start with that if we can. If it's not feasible, a hard error can also work (I would say though that we should upstream PRs to any crates we break).

Another small thing I noticed is that the error message links to the Nomicon section on variance, but it would be ideal to link to a tracking issue or something describing this issue in particular.

[traviscross]

To add on to what tmandry, said, in our discussions we did feel that the approach taken in this PR is generally the right way forward, and we're happy to see this progress so as to help clear the way for arbitrary_self_types and derive_coerce_pointee.

cc @rust-lang/lang

[BoxyUwU]

@tmandry I do expect it to be possible to FCW this. We can likely do something hacky around to fully emulate the fix (but as a lint), but if that doesn't work out all the regression we found were relatively "simple" cases that can probably be taken advantage of (if need be) to lint a subset of the actual cases we'd break with this PR

edit: see compiler-errors' comment, I'm not so convinced this will be possible to FCW anymore and will likely investigate improving the diagnostics here. I've already filed PRs to the affected crates to migrate them over to a transmute to avoid the breakage if this lands

[compiler-errors]

I was thinking earlier that it may be possible to implement a lint to detect, but it seems to me that MIR borrowck is not equipped to implement such a lint.

Specifically, it seems near impossible to answer whether a region outlives constraint (like, 'a: 'b) would not hold in a way that doesn't actually commit to that constraint, at least not without tons of false positives based on how NLL computes lower bounds for all of the regions it deals with in the MIR.

To fix this would require some significant engineering effort to refactor how NLL processes its region graph to make it easier to clone and reprocess with new constraints.

[tmandry]

Thanks for the feedback everyone. I was the one primarily pushing to avoid unnecessary breakage in this case. That is conditional on the feasibility and maintainability of a solution, and @compiler-errors makes a compelling case that it would be neither. No one thought we were proposing changes to MIR borrow check! @BoxyUwU may be able to confirm/deny if that is what she had in mind.

I think we've heard enough to convince me that this is not a path worth going down now. It sounds like the lint mitigation was based on some faulty assumptions. Upon reflection, I agree with @BoxyUwU that we should have consensus within the lang team to walk this safe/unsafe difference back over an edition before releasing it. Deciding this is infeasible or a maintenance burden, or being convinced (as others seem to be) that the breakage is very minimal, would be enough as well.

In this particular case I think the breakage is unfortunate but workable, so I am okay to move forward. I don't speak for the rest of the lang team in this comment, but no one spoke up about wanting to avoid the breakage as much as I did.

@Manishearth You mentioned an FCW; my understanding has been that an FCW in this case is infeasible. I do agree with your assertion that this would make unsafe reviews harder.

IMO this is a pretty drastic new behavior to have to introduce just to avoid crate breakage from something we only recently started to allow

@compiler-errors That wasn't my understanding from the discussion, which included a 6yr old version of diesel. Was that relying on this as an unstable feature until recently, or what am I missing?

[compiler-errors]

@tmandry: I guess I was a bit misled when I said we only recently started to allow this behavior. What I should have said is that we only recently made this behavior more relaxed beginning with #113262.

That PR made casts where nothing but the lifetime was changing work, i.e. expr as *const dyn Tr + '_ as *const dyn Tr + 'static even if expr's type doesn't outlive 'static. That's why most of the regressions are from ~1 year ago or so.

But there are rare cases where this behavior coincidentally worked when there were more than 1 casts chained together and between the two casts more than the lifetime changed in the casted type. That old diesel one is an example. Boxy and I worked out a few examples where this was allowed even pre 1.75 which is when that linked PR landed.

This is the moral equivalent of the diesel example:

trait Tr {}

struct Foo {
    ptr: dyn Tr + 'static
}

fn foo<T: Tr>(x: &dyn Tr) -> *const Foo {
    let x = x as *const dyn Tr as *const Foo;
    x
}

And this is the moral equivalent of the brainfuck one:

trait Tr {}

fn foo<T: Tr>(x: &dyn Tr) -> *mut (dyn Tr + 'static) {
    x as *const dyn Tr as *mut dyn Tr
}

Both of these examples worked before 1.75 because they were emitting something that was a non-trivial pointer cast. But they still are definitely violations of the vtable validity here.

---

I will note that we're still really funny with what we allow when doing raw pointer casts of wide pointers. For example, this errors today:

trait Tr {}

fn foo<T: Tr>(x: &dyn Tr) -> *const (dyn Tr + 'static) {
    x as *const dyn Tr
}

i.e. it was not fixed by #113262. You could argue that @BoxyUwU's PR here is making the language more consistent by always enforcing the correct lifetimes in wide pointers along these lines :)

[Veykril]

@Veykril Could you weigh in on the technical feasibility of both detecting this new class of unsafe-but-only-when-lifetime-changing as casts, using r-a's understanding of how Rust code works, and emitting the right highlighting suggestions to editors to identify this for users?

given rust-analyzer is still completely oblivious to lifetimes in our IRs I'd say very difficult. Though either way that feature is only meant to show what would be erroring if the unsafe block was missing, which reading from this, the as cast wouldn't even do in the first place? (as you said it is more of a new class of unsafe)

[traviscross]

On the substance here, we'll obviously take this back up on Wednesday. My estimate is that we're rather likely to readopt the original plan and to "stay the course" on that. We very nearly went that way last Wednesday.

Speaking for myself, I think what's being done here, in terms of making PRs to the affected public projects and working out a diagnostic that guides people in the right direction, is what we need to do, and combined with the soundness arguments we've been making, is sufficient to justify and support this change. I appreciate -- and I know we all on lang appreciate -- the work @BoxyUwU did to make those PRs and is doing to put this all together. It's a nice additional benefit that we apparently only allowed the most likely kinds of this semi-recently and that this probably does make the language more consistent in the way that CE mentioned.

[RalfJung]

making PRs to the affected public projects and working out a diagnostic that guides people in the right direction

By "diagnostic" here you refer to the hard error that shows in code this breaks?

@BoxyUwU @compiler-errors how practical is it to give a dedicated diagnostic for this case?

[traviscross]

By "diagnostic" here you refer to the hard error that shows in code this breaks?

That's right.

[Manishearth]

@Manishearth You mentioned an FCW; my understanding has been that an FCW in this case is infeasible. I do agree with your assertion that this would make unsafe reviews harder.

@tmandry

Maybe I'm missing something, and part of the issue with this thread is that it's not always clear what changes specifically are being talked about.

But for the proposal on the table that is:

• Forbid some things entirely in as casts in safe rust
• Allow them in unsafe rust but unsafe rust writers need to worry about the precise usage pattern to avoid/trigger UB

It should be instead possible to forbid those things entirely in all as casts, and have an FCW/edition/whatever for them, since if it is possible to forbid a thing it should be possible to use the same mechanism to lint it.

If the FCW for forbidding such casts is impossible to write, then I ask: how are you going to forbid the operation completely in safe Rust as is the lang team's proposal?

---

@traviscross

On the substance here, we'll obviously take this back up on Wednesday. My estimate is that we're rather likely to readopt the original plan and to "stay the course" on that. We very nearly went that way last Wednesday.

Could you explicitly write down the "original plan"? This thread is rather self referential, the linked comment doesn't specify the plan, it refers to a previous comment which also isn't fully clear.

[compiler-errors]

If the FCW for forbidding such casts is impossible to write, then I ask: how are you going to forbid the operation completely in safe Rust as is the lang team's proposal?

Because it amounts to literally running all of borrowck twice. It's not impossible to write, but it's prohibitively difficult to do and would require quite a lot of refactoring. Straight up denying it amounts to a few lines of change, as implemented in the PR today.

[traviscross]

Could you explicitly write down the "original plan"?

It's what I went on to say. We do the breaking change along with a diagnostic on the error that tries to point people in the right direction and we make PRs to affected projects.

[Manishearth]

@traviscross there are multiple possible breaking changes in this thread, and multiple lints. I don't actually think anyone has precisely defined what is happening so far all in one place, instead referencing previous comments with wording like "the breaking change".

[BoxyUwU]

The breaking change is that casting *const dyn Trait + 'a to *const dyn Trait + 'b will now require 'a: 'b to hold, whereas previously these lifetimes were allowed to be entirely unrelated.

[Manishearth]

Hmm, from what I recall that specific example is possible to lint against in a late lint pass. There's a lot of talk of MIR borrowck when talking about feasability, maybe I'm missing something, but from clippy experience that seems implementable?

[BoxyUwU]

It's not just about that specific example, for the general case you need borrowck to check things like *const dyn Trait + '_ being cast to *const dyn Trait + '_ doesnt wind up doing something equivalent to that example.

[Manishearth]

I mean, yes, I understand it's not a simple AST match, but ultimately those are both TyKind::Dynamics, which does still retain the Region`. I can imagine there are a lot of corner case Regions that are not easily compared, but I was under the impression it's possible for the common cases. From digging deeper it seems like it's more complicated than that.

An FCW with a lot of false positives or false negatives is potentially still worth trying, though, if they don't end up triggering often.

[RalfJung]

@Manishearth The only part of the compiler that actually knows lifetimes is borrowck. So nothing outside borrowck can implement this lint as there's no way to actually get hold of the two lifetimes that have to outlive each other.

Inside borrowck, the way the hard error is implemented is to add this new constraint to the big sea of constraints that is gathered up during borrowck. Borrowck never explicitly materializes concrete lifetimes, all it does is gather constraints and then check if the constraint system is satisfiable. That makes it impossible to lint since when borrowck fails, it is entirely unclear whether it would have succeeded without that extra constraint.

[Manishearth]

Ah, I see. I was under the impression some of this information was available after MIR borrowck, but looking through clippy we don't actually do that. Fair enough.

(I may be operating on a model of things from before MIR borrowck)

[traviscross]

We talked about this on the lang call today, and we'd like to unblock this in favor of proceeding with the original plan, which is that we plan to accept the breaking change as originally proposed in this PR subject to PRs being made to affected projects (already done) and an attempt being made to provide a diagnostic along with the hard error that will point people in the right direction on this as best as possible.

When this PR is ready, along those lines, please renominate it for us, and we'll propose lang + types FCP.

@rustbot labels -I-lang-nominated +I-lang-radar

[BoxyUwU]

I've pushed a first attempt at some special cased diagnostics for this breakage, it links to #141402 which could probably do with some work.

[BoxyUwU]

Oh another update: the metrics crate which was responsible for the vast majority of regressions has been fixed and had the fix backported to older versions too. I think may was also fixed in a non-major release. So both may and metrics users broken by this should be able to cargo update to be unbroken