Specify temporary lifetime extension through expressions

[dianne]

Reference PR for rust-lang/rust#146098. This includes a reworked definition of extending expressions with the aim of expressing the new semantics more uniformly.

[dianne]

A major change here: an extending expression is now any expression that preserves lifetime extension, defined non-inductively. I found this helps with generalizing the definition beyond let statement initializers, but I also often found myself having to refer to an expression being "extending when its parent is extending"; that's are now just an extending expression.

[traviscross]

Yes, this greatly widens the definition of an extending expression, putting more load on the other rules.

[dianne]

Something slightly weird here: the last expression of a const block morally should be here, but it'd be a bit messy to have to exclude it from the rule for blocks lower down. Given that this definition of extending expressions doesn't care about where scopes are extended to, it shouldn't be a semantic issue, but it might warrant reformatting and/or an admonition.

[traviscross]

Let's see the admonition for this (with one or more examples).

[dianne]

I've handled this by making const block tail expressions no longer be defined as "extending" (along with let statement initializers and static/const item bodies). That way, there's less room for confusion, hopefully. I've also added an example of extending through a const block tail expression to illustrate it (taken from #2041)

[dianne]

I struggled a bit with how to express this; in its current form it's a bit of a hack. Ideally, I feel like it wouldn't need to be a separate rule or to refer to the definition of scope nesting, but it's working around something subtle: by ensuring that expressions' temporary scopes are only extended by their scope-ancestors, we can work around const blocks having parent expressions that (to my understanding) shouldn't be considered ancestor scopes of the const block's body; temporaries extended by const blocks are extended to the end of the program¹. Maybe there's a simpler way to express this, and regardless it could probably use an admonition.

I do think that some sort of "extended by" or "extended through" or "extending based on" relation is necessary though, regardless of how exactly we choose to define/present it. I feel there's too much ambiguity if we can't precisely associate expressions we're extending the temporary scopes of with the scopes they're being extended to.

Footnotes

1. This PR doesn't make all the changes needed to iron that out, but see Further specify temporary scoping for statics and consts #2041. ↩

[traviscross]

Let's see the admonition that you have in mind here (with one or more examples). Let's also add rules to define what it means exactly to "extend to", "extend through", etc.

[traviscross]

In particular, what it means to "extend through" a parent scope is going to be worth careful explanation and one or more examples in an admonition.

[dianne]

I've fully reworked the formulation of the upward walk from & expressions and added examples.

[dianne]

Some additional examples would probably be good. Maybe it would help to have one where temporaries are extended through a block but not to the end of a statement? That could also be used as a compile_fail example.

[dianne]

Maybe it could also use some additional text (or even an admonition?) to make clear the interaction with if block scopes and Rust 2024's tail expression scopes. I'm not sure exactly how much explaining is needed for that, though.

[traviscross]

This is all rather subtle. Erring on the side of more examples and explanation will be better here.

[dianne]

I haven't added additional examples for this case of lifetime extension yet, but I fleshed out the first one quite a bit to walk through what's going on. I removed the second in the process, but I'll think on how best to readd it later (along with negative examples).

[dianne]

I wonder if there's a way to cut down on the duplication here. I felt these rules were necessary to be precise about where temporaries' scopes are extended to, but having them in the introduction to the lifetime extension feels necessary too.

[traviscross]

Agreed they felt necessary here.

[traviscross]

Thanks @dianne; will have a look.

[traviscross]

@ehuss and I looked carefully through this today. Some thoughts.

After staring at this awhile, I see why this makes it easier to express the core idea of rust-lang/rust#146098 -- that the relative drop order within an expression should be consistent regardless of where that expression is. Expressing the rules inductively for defining an extending expression ends up pushing against this.

So it does, I think, make a deep sort of sense to drop the inductive approach, as this PR does.

At the same time, the inductive approach was carrying a lot of load. The reader could visualize walking down an expression, outside to inside, at each step checking whether it was still an extending expression. Then there was essentially one key rule that leaned on that:

The operand of an extending [borrow] expression has its [temporary scope] [extended].

What this PR does, in dropping the inductive framing, is to widen the definition of an extending expression significantly, and thereby put a lot more load on the rules that follow to narrow this back down. That's where it ends up getting a bit subtle. With this approach, both @ehuss and I found building intuition and checking the correctness of the rules to be more difficult. So, in dropping the inductive approach, I think we have some work to do in order to recover the same level of clarity.

Adding some !EXAMPLE admonitions that walk through applying these rules would help and be a good start here. It'll also be worth more carefully defining what it means to "extend to" or "extend through" something. The bit where we define some things as "extending through" but then "clamp" them so as to only "extend to" with the final rule ("A temporary scope extended through the scope of a non-extending expression is [extended] to that expression's [temporary scope].") is another bit of subtlety that it might be worth finding another way to express.

I'll leave some other thoughts inline.

[traviscross]

This rule is particular is one where I wonder how we might do better. The idea, currently, is that the rules above define when a temporary scope would be "extended through" the scope of an expression, and then this rule says, well, if it's a non-extending expression, then we only "extend to" it.

[dianne]

I've reworked this to avoid the hack of using scopes to express hierarchy entirely.

[traviscross]

It's a bit subtle that, after having defined above what is an extending expression, that this rule then doesn't use that definition at all. It makes sense -- all operands of any borrow expression are extending. But it'll be worth adding an admonition under each of these rules that elaborates the rationale and gives one or more examples.

[dianne]

I've reworked this: now it appears before the definition of extending expressions. It's difficult to provide admonitions and examples before defining all the relevant terms, but I've tried to connect the rules together and provided some detailed examples towards the end.

[dianne]

I'll try and address individual points soon, but first one higher-level note that I'll try to use as guidance when restructuring:

At the same time, the inductive approach was carrying a lot of load. The reader could visualize walking down an expression, outside to inside, at each step checking whether it was still an extending expression.

This PR's approach still has a visual intuition to it, so it should be presented in a way that makes that clear: instead of walking down an extending expression to check whether an &'s operand is extended, you walk up from an & and check whether you're in an extending expression at each step to determine the temporary scope of the &'s operand. Possibly a separate inductive definition would be useful for making that clear? Along with examples, of course. I was worried defining too many things all at once would make it difficult to follow, but examples should help.

[traviscross]

This PR's approach still has a visual intuition to it, so it should be presented in a way that makes that clear: instead of walking down an extending expression to check whether an &'s operand is extended, you walk up from an & and check whether you're in an extending expression at each step to determine the temporary scope of the &'s operand. Possibly a separate inductive definition would be useful for making that clear? Along with examples, of course.

Yes, exactly. This upward walk is what I'd been expecting but it has to be teased out a bit from the presentation here. Hopefully making that more clear will make the rules here more clear.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[dianne]

I did a pass to clean up definitions, reframe "extending based on expressions" as a property of & expressions (rather than of extending expressions or let statements), and add clarification and examples. Some parts are still awkward, unfortunately. I've left review comments on a couple things in particular I'm not happy with yet.

[dianne]

Two things here:

• I think it's helpful to start destructors.scope.lifetime-extension.exprs with a straightforward explanation of what all the definitions are for. Otherwise, I found that either too many definitions are needed before any of them can be explained, or the explanations have to refer to terms that haven't been defined yet. For the sake of directness, I moved destructors.scope.lifetime-extension.exprs.borrows to the start, but that meant having to refer to terms that are defined later. A possible alternative would be writing a new destructors.scope.lifetime-extension.exprs.intro rule, but when I tried that it felt redundant.
• I don't love the term "extended scope", but I wasn't able to think of anything better, either in terms of naming or in terms of reframing to avoid defining it at all. I feel like it could probably also use some additional clarification but I'm still thinking of a good admonition to add for it.

[dianne]

Three things here:

• To simplify the definitions of extending expressions and extended scopes, I've made let initializers, static/const items, and const block tails no longer extending. This might hurt the intuition of extending expressions a bit though.
• I added the definition of "extending parent" when reworking the upwards walk. This keeps it rigorous without hacks (the parent of an expression isn't defined elsewhere, and going through the parent of a scope to save words is confusing). It also also makes it clear that extending expressions are all subexpressions and thus have parent expressions. But spending words on it feels a bit awkward too, either adding bloat or making things convoluted (or both).
• Extending expressions are necessary to define extended scopes, but extended scopes are necessary to justify the definition of extending expressions. As such, I felt a need to connect this to the surrounding sections, but I don't love how it turned out. Maybe there's a better way to handle that.

[dianne]

Following up on making "extending parents" make more sense and making it clear that "extending expression" isn't a classification of expressions but a classification of subexpressions, maybe it would be worth renaming "extending expression" to "extending subexpression"? I think it makes more sense from a spec point of view, but it's more awkward to read and write (and it means having to adapt language elsewhere), so I haven't gone through with it.

[traviscross]

It has some appeal, but I similarly see the problems you mention. If one says "extending subexpression", then one kind of wants the parent to be an "extending expression" (in the same way as "header", "subheader", etc.), but that's probably confusing.

[dianne]

I think this may need a terminology update, a carve-out in destructors.scope.lifetime-extension.exprs, or at least clarification on what "extended temporary scope" means. In particular, if you have an expression like

{ &*&temp() }

it's important that this rule takes precedence, so that temp() lives past the block. destructors.scope.lifetime-extension.exprs.borrows currently is worded to be the definitive lifetime of borrow operators' operands, but if only that was considered, temp() would be dropped at the end of the tail expression.

[dianne]

This is partially addressed by the new example in 6536b3b. I think it'll still also want some reformulation or clarification of lifetime-extension.sub-expressions to make really make sense, though.

[traviscross]

Doesn't it almost feel as though the things in this rule should be rolled in to "extending based on expressions"? E.g.:

The extended scope of an expression is defined in terms of extending expressions and their extending parents. An extending expression is an expression which is one of the following... the indexed expression of a tuple indexing expression, the extending parent of which is the tuple indexing expression.

Or is there some important distinction we're trying to encode here in these being handled separately?

[dianne]

There's a couple distinctions.

First, except for borrow expression operands, none of those subexpressions are extending. For instance,

let x = (&PrintOnDrop("temporary dropped")).0;
println!("next statement");

prints "temporary dropped", then "next statement". Intuitively, since the result of indexing is evaluated and then copied into x, the temporary doesn't need to be extended past its use, so we don't extend it. The purpose of lifetime-extension.sub-expressions is to handle reborrows, borrows of projections, etc.: if we'd instead written

let x = &(&PrintOnDrop("temporary dropped")).0;
println!("next statement");

we'd be reborrowing the temporary's field rather than evaluating it, so we would extend the temporary, so it'd print "next statement", then "temporary dropped". Extending subexpressions generally should only be those where if you substituted a borrow expression in, it's syntactically obvious that the borrow will need to be live to use the result of evaluating the extending parent expression. Maybe there's a way to express that intuition productively in an admonition somehow?

Second, lifetime-extension.sub-expressions also applies when extending based on patterns:

let ref x = PrintOnDrop("temporary dropped").0;
println!("next statement");

prints "next statement", then "temporary dropped". Because x is borrowing from a place based on the temporary, the entire temporary is extended. The example in lifetime-extension.sub-expressions illustrates this in a bit more detail.

[traviscross]

I don't fully understand how to apply this rule in the same way that rustc does. I put together a series of examples and mechanically analyzed them according to the rules in this PR. Can you explain the behavior of the last two examples in light of the behavior of the ones that precede them?

Playground link

[traviscross]

Setting that aside, and setting aside the matter of the interaction with "extending based on patterns" (which as we discussed on Zulip, might end up needing to be restructured eventually given how it would interact with a binding version of super let or super let in), it feels to me as though these are properly a kind of extending expression but that, as we work outward, we're carrying a flag indicating whether we last stepped through a "full" extending expression or through a more "limited" extending expression, and the value of that flag affects our behavior, e.g. when we get out to the let initializer.

(To the degree that's not true, maybe it's worth considering whether it should be.)

[dianne]

My notes on each example, per the mental model I was aiming for with that change (I'll address the limited extending model after):

1. let _ = (&e.log(1)).0; //~ Not extended.
   Our analyses agree here.
2. let _ = &(&e.log(2)).0; //~ Extended.
   Our analyses agree here.
3. let _ = &(&(e.log(2),).0).0; //~ Extended.
e.log(2) here isn't extended; instead, it's immediately moved into the tuple temporary, which is extended. I believe its scope is that of the tuple expression, per destructors.scope.operands. I'd also add another step: the reason (&(e.log(2),).0).0 uses its extended scope as its temporary scope is because it's the operand of a borrow expression. Otherwise, our analyses agree.
4. let _ = &[(e.log(2),).0][0].0; //~ Extended.
   In this case, the temporary that gets extended is the array given by evaluating [(e.log(2),).0]. A place based on it, array[0].0, is borrowed, and since we want that borrow to be live for the block, we extend the array's lifetime.
5. let _ = &[(e.log(2),).0]; //~ Extended.
   As with 4, the temporary that gets extended is the array given by evaluating [(e.log(2),).0].
6. let _ = &[(&e.log(1)).0]; //~ Not extended.
   As with 4 and 5, the temporary being extended is the array. The key here is that we're not borrowing from the e.log(1) temporary in the final value; we evaluate (&e.log(1)).0 to something like LogDrop(_e1, 1) where _e1 is a reborrow of e and then move that value into the array. As such, e.log(1) does not need to be extended; this is why tuple indexing expressions are not extending parents.
7. let _ = &(&e.log(1),).0; //~ Not extended.
   This one we should extend according to the new model we're discussing. e.log(1) should have the tuple expression's temporary scope, which should be extended. Indeed, it's currently an error to use the result because it borrows from the dropped e.log(1) temporary. I don't think my compiler PR handles this yet either, but I'd like for it to work.
   The way rustc does this is when it extends (&e.log(1),).0 it applies lifetime-extension.sub-expressions to extend the place(s) it's based on as well (so (&e.log(1),)) and then stops.

---

For the limited extending model, I think there's a conceptual reason to keep them distinct. Extending parent expressions represent things like constructors and conditionals that build (or select) a value and are guaranteed to contain (or evaluate to) their extending subexpressions. Projections like .0 are sort of a dual to that: if we want to extend the lifetime of a projected place, we have to extend the base place too, thus we extend the indexed operand. Actually, I think under this PR's model, we can clean it up and get & out of the "sub-expressions" list so it's just the base places of place expressions. I'll experiment with that on the compiler side of things and update this PR if it works out.

[dianne]

I wonder what the right thing to do is in cases like

let x = [&temp(), local][1];
let y = &[&temp(), local][1];

I see a few options:

1. Under rustc's current model (including under Temporary lifetime extension for blocks rust#146098 currently), neither let statement extends its temp(). This is safest in a way: this means we're not extending unused temporaries to the end of the block. Of course, it's still possible today to create unused extended temporaries, e.g. with let _ = &temp();.
2. Under this PR after the latest update, I think only the definition of y would extend its temp(), since the array would have an extended temporary scope and temp() would have the array's temporary scope as its temporary scope. In the case of x, the array wouldn't have an extended temporary scope because the result of indexing is used as a value.
3. Under a model where indexed expressions are limited-extending, I could possibly see both statements extending their temp()s, because in either case we can see that the temp() could end up referenced in the final value. I'd have to think through what the exact rules would be to end up with that result.

[traviscross]

Makes sense. Regarding 3..=6... oops, yes, of course -- I must have been staring at too many variations for too long. Interesting about 7.

Revised analysis of the interesting cases:

Playground link

Regarding the sub-expressions rule:

If a [borrow], [dereference][dereference expression], [field][field expression], or [tuple indexing expression] has an extended temporary scope, then so does its operand. If an [indexing expression] has an extended temporary scope, then the indexed expression also has an extended temporary scope.

In "extending based on expressions", we're defining expressions to have an extended scope even if that extended scope is never used. E.g.:

r[destructors.scope.lifetime-extension.exprs.let]
The extended scope of the initializer expression of a let statement is the scope of the block containing the let statement.

For sub-expressions, then, it seems we have to account for that somehow.

[dianne]

In "extending based on expressions", we're defining expressions to have an extended scope even if that extended scope is never used.

For sub-expressions, then, it seems we have to account for that somehow.

This is a terminology problem. I couldn't think of a better term than "extended scope" at the time, but there's meant to be a distinction. Every expression has an extended scope. Borrow expressions' operands use their extended scope as their temporary scope; as such, they're said to have an extended temporary scope (though it looks like I forgot to add that definition). The initializer expression of a let statement with an extending pattern also has an extended temporary scope.

I can think of a few things that might help:

• Actually defining "extended temporary scope". That term is a holdover from the stable version of the temporary lifetime extension section, where it was more obvious what it meant. It might still be obvious enough here if we can get rid of the other too-similar terms, and under the definition I proposed above it might be confusing that the "extended temporary scope" of an expression can sometimes be its normal temporary scope.
• Removing the term "extended temporary scope". sub-expressions is e.g. saying that the tuple operand expression of a tuple indexing expression has the same temporary scope as the tuple indexing expression. We don't need a notion of "extended temporary scope" to express that.
• Renaming the term "extended scope". The term I used in the compiler is "extended_parent scope" to match with the notions of "parent scope" and "var_parent scope" used there. That feels too close to "extending parent" though, maybe something like "extended ancestor scope" would be fine? I'm not sure.
• Removing the term "extended scope". e.g. a previous iteration of this PR defined the "extending ancestor" of an expression as the closest non-extending expression above that expression, then defined the scopes of borrow expressions' operands and super temporaries in terms of its temporary scope. It felt messier, so I ended making rules to determine its temporary scope directly (as the "extended scope" of an expression).

With this in mind, there isn't intended to be any ambiguity or a need to determine precedence between exprs and sub-expressions. The rules should be written in a way that makes everything clear. They just haven't quite gotten there yet.

[traviscross]

We looked through this in the lang-docs office hours call today. The recent changes here are great. Big improvement in clarity. Thanks to @dianne for those.

We talked about how this might benefit from an introduction (perhaps in an admonition) to the "extending based on expressions" section that tries to give some intuition -- a reading guide -- to a reader coming into this section.

I'm going to give this another careful read through, and @ehuss is planning to run some tests with the PR to make sure the interaction of this with the 2024 edition change is clear, but overall, this seems really close.

[traviscross]

It's maybe not what we want to do right now, but it occurred to me that these and the other rules could probably be usefully presented in the manner of inference rules, e.g. (and making no assertions about correctness of this sketch):

enum Scope {
    Block,
    Stmt,
    // ...
}
struct S(scope: Scope, ext_scope: Scope);

/// Borrow expressions
S(x, x) ⊢  $expr
----------------
S(s, x) ⊢ &$expr

/// Array expressions
S(s, x) ⊢  $expr
----------------------
S(s, x) ⊢ [$expr, ...]

/// Function calls
S(s, s) ⊢       $expr
---------------------------
S(s, x) ⊢ $path($expr, ...)

/// Let initializers
S(Stmt, Block) ⊢           &$expr
---------------------------------
S(Stmt, Block) ⊢ let ... = &$expr

// Etc.

When @Nadrieril and I were working on match ergonomics, we ended up finding it rather helpful for analysis to present the rules in this manner. See, e.g. this document for a discussion of notation.

@Nadrieril ended up writing a cool web tool for doing analysis and comparisons based on this kind of notation.

It's similarly been occurring to me here that it would be helpful to have a tool that encodes the formal rules in software and walks through the analysis of an expression. (Of course, one option is always to do something like that in the compiler with an approach similar to #[rustc_capture_analysis]).

[dianne]

I'm pretty familiar with inference rules; if it would help to discuss them in that way, I could write them out for different variants of lifetime extension (e.g. the stable version, the compiler PR version where exprs.other uses the enclosing scope of the extended parent, the version we've been talking about where exprs.other uses the temporary scope of the extended parent, etc.).

[traviscross]

That would definitely be helpful. I could see including that directly in the Reference as well.