Discuss variable priorities and shadowing

[mihnita]

The PR #305 trying to document the hoisting decision introduced the wording that has effect on the shadowing resolution.

To resolve the value of a Variable, its Name is used to identify either a local variable, or a variable defined elsewhere.
If a local variable and an externally defined one use the same name, the local variable takes precedence.

That was not part of a discussion, and the decision is debatable.

[mihnita]

The PR discussion thread, for convenience:

---

@stasm stasm 8 days ago
This implies that name shadowing is possible. My current thinking is that it should be OK, and I see use-cases for it. However, I don't recall an explicit discussion on the subject. It might also make sense to start with a stricter design and disallow shadowing at first.

---

@eemeli eemeli 8 days ago
I rather think that local variables must be allowed to shadow externally provided named arguments.

For instance, I could well imagine translation systems with the following attributes:

Translators are able to define additional local variables for a message.
A developer may pass the same basket of arguments to multiple messages, not all of which use each of them.
If shadowing were not allowed, then a combination of the above could result in a runtime error which would be difficult to discover ahead of time.

---

@mihnita mihnita 23 seconds ago
I think that this part of the change (shadowing) is a bit of an overreach.
It is unrelated to the hoisting problem, and it was not commonly agreed.

In general I don't think it is a good idea to allow translators to override a developer decision.
So "Translators are able to define additional local variables for a message." is a cons in my book, not a pro.

And "A developer may pass the same basket of arguments to multiple messages, not all of which use each of them." does not look like something related to shadowing.

There is a reason many linters warn about shadowing.

"If shadowing were not allowed, then a combination of the above could result in a runtime error which would be difficult to discover ahead of time."

Not necessarily.
One can say that translators can't define variables (and that is a good thing).
It is an error, but the decision between runtime error and simply ignore the conflicting local variable is implementation dependent.
We should not use that as an argument for what the standard should do.

But TLDR:

I don't see valid use-cases for it. In fact the proposal we had used different sigils for local variables (macros at the time) and arguments.
I don't recall an explicit discussion on the subject.
It might also make sense to start with a stricter design and disallow shadowing at first.

[mihnita]

Adding to it a bit: in one of the discussions Eemeli introduced the idea that we can have a "map of variables" passed at parse time, in addition to the one we pass at formatting time.

For example:

pattern = '
    let $foo = {$count :number} // LOCAL VARIABLES
    {Hello {$foo} and {$bar}!}
'

mf = MessageFormat2.parse(pattern, { 'foo': 42, 'bar': 'success', 'baz': 'this works' } ) // VARIABLE MAP 1
result = mf.format({ 'foo': 101, $bar: 'fail', 'count': 12345 } ) // VARIABLE MAP 2

So when the messages uses $foo and $bar we need to decide not only local variables vs variables defined elsewhere, but also the priority between the various "elsewhere"

I remember that in a previous discussion we discussed having several "bags of arguments", and the simple answer was to merge the bags before passing them to message format.
So the decision would be with the developer what overrides what, without complicating the standard.

But anyway, if we allow for more bags, we need to decide the priority.

[mihnita]

To summarize my arguments against this functionality:

---

1. There is no useful functionality added by allowing variable overrides

This is probably already reasonably well captured in previous comments, so I will not add any details.

---

2. Adds complexity to the spec and implementations

For example we previously discussed the risk or circular dependencies in definitions:

let b = {$a ...}
let c = {$b ...}
let a = {$c ...}

The mitigation was to not do hoisting, voted for by all people in that thread:
(zbraniecki, macchiati, eemeli, stasm, mihnita)
#297

But with this change we reintroduce the risk of circular dependencies:

let a = {$foo ...}
let b = {$a ...}
let a = {$b ...}

It is not hoisting, the declaration of a in line 3 can see the declaration of b in line 2, which can see the declaration of a in line 1.
Unless the definition of b actually "sees" the definition of a in line 1. So that definition is no completely shadowed by line 3, as a "left over" is captured in the definition of b.

In other words, if I do let a = {42} let b = {$a} let a = {13} let c = {$a} {The values are {b} and {c} !}
What is the result?
It it is "The values are 13 and 13 !", this is hoisting-like, and allows circular dependencies.
And it if is "The values are 42 and 13 !", so the first definition is not "fully shadowed".
(behaves like macros in C :-)

So the spec would have to explain what happens in case 2, how to avoid circular definitions (if we shadow 100%).
And implementations will have to follow that spec.

That is adding complexity to the spec / implementations, for no good benefit
("no benefit" is a personal judgement, see item 1, we can debate it)

I'm fine to add complexity as a price for something useful (balancing price vs benefit).

And this PR does not seem to explain the behavior in the above cases.

---

3. People are used to it

If memory serves me, I think this was mentioned in the meeting 2 weeks ago.

If not, I apologize, this would be a straw-man, please ignore.

So, would people expect it, because it is common?
Yes and no.

a. Many linters warn about shadowing.

So it is not necessarily a good thing.

b. Ant properties are immutable

See https://ant.apache.org/manual/Tasks/property.html:
<property name="foo.dist" value="dist"/>

And I've seen huge ant files implementing complex functionality despite this "limitation"

So it is not unheard of, and it is not that limiting.

c. In Rust you let a = 42, and it is immutable by default.

To make it mutable you need to say let mut a = 42
And in code the review the first question will probably "why does this have to be mutable?"
So the same idea from item a.: this is not a good thing, needs to serve a purpose.
Even the keyword is let, like we have.
So Rust does not try to accommodate some expectation of mutability.

d. Even in JS let is not like what we do here

In JS let can be updated but not re-declared

let a = 43 // OK
a = 13 // OK
let = 101 // Error

But with this PR we allow re-declaration, so we are even more "lax" than JS
(because for us declaration is initialization, which is a good thing)

[macchiati]

For my part, I see no reason why variables need to be mutable (aka mut). It is not a particular imposition on the writer to use a slightly altered name. The exposition and implementation are both simpler.

…

On Thu, Jun 15, 2023 at 3:01 PM Mihai Nita ***@***.***> wrote: To summarize my arguments against this functionality: ------------------------------ 1. There is no useful functionality added by allowing variable overrides This is probably already reasonably well captured in previous comments, so I will not add any details. ------------------------------ 1. Adds complexity to the spec and implementations For example we previously discussed the risk or circular dependencies in definitions: let b = {$a ...} let c = {$b ...} let a = {$c ...} The mitigation was to not do hoisting, voted for by all people in that thread: (zbraniecki, macchiati, eemeli, stasm, mihnita) #297 <#297> But with this change we reintroduce the risk of circular dependencies: let a = {$foo ...} let b = {$a ...} let a = {$b ...} It is not hoisting, the declaration of a in line 3 can see the declaration of b in line 2, which can see the declaration of a in line 1. Unless the definition of b actually "sees" the definition of a in line 1. So that definition is no completely shadowed by line 3, as a "left over" is captured in the definition of b. In other words, if I do let a = {42} let b = {$a} let a = {13} let c = {$a} {The values are {b} and {c} !} What is the result? It it is "The values are 13 and 13 !", this is hoisting-like, and allows circular dependencies. And it if is "The values are 42 and 13 !", so the first definition is not "fully shadowed". (behaves like macros in C :-) So the spec would have to explain what happens in case 2, how to avoid circular definitions (if we shadow 100%). And implementations will have to follow that spec. That is adding complexity to the spec / implementations, for no good benefit ("no benefit" is a personal judgement, see item 1, we can debate it) I'm fine to add complexity as a price for something useful (balancing price vs benefit). And this PR does not seem to explain the behavior in the above cases. ------------------------------ 1. People are used to it If memory serves me, I think this was mentioned in the meeting 2 weeks ago. If not, I apologize, this would be a straw-man, please ignore. So, would people expect it, because it is common? Yes and no. a. Many linters warn about shadowing. So it is not necessarily a good thing. b. Ant properties are immutable See https://ant.apache.org/manual/Tasks/property.html: <property name="foo.dist" value="dist"/> And I've seen huge ant files implementing complex functionality despite this "limitation" So it is not unheard of, and it is not that limiting. c. In Rust you let a = 42, and it is immutable by default. To make it mutable you need to say let mut a = 42 And in code the review the first question will probably *"why does this have to be mutable?"* So the same idea from item a.: this is not a good thing, needs to serve a purpose. Even the keyword is let, like we have. So Rust does not try to accommodate some expectation of mutability. d. Even in JS let is not like what we do here In JS let can be updated but not re-declared let a = 43 // OK a = 13 // OK let = 101 // Error But with this PR we allow re-declaration, so we are even more "lax" than JS (because for us declaration is initialization, which is a good thing) — Reply to this email directly, view it on GitHub <#310 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMAP6Y5DFDFNW2LFTN3XLOA4RANCNFSM6AAAAAARPNDAZI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[eemeli]

From @mihnita: (note: example edited to fix syntax)
In other words, if I do

let $a = {42}
let $b = {$a}
let $a = {13}
let $c = {$a}
{The values are {$b} and {$c} !}

What is the result? It it is "The values are 13 and 13 !", this is hoisting-like, and allows circular dependencies. And it if is "The values are 42 and 13 !", so the first definition is not "fully shadowed". (behaves like macros in C :-)

So the spec would have to explain what happens in case 2, how to avoid circular definitions (if we shadow 100%). And implementations will have to follow that spec. [...]

And this PR does not seem to explain the behavior in the above cases.

Presuming that this is in reference to PR #381, that proposal currently includes this:

To resolve the value of a variable, its name is used to identify either a local variable, or a variable defined elsewhere.

If more than one declaration binds a value to the same name, or if an externally defined variable and a declaration use the same name, the resolved value of the most recent declaration preceding the variable is used.

Taking this one step at a time:

1. To format the pattern, we need to resolve the expression {$b}, which contains a variable $b.

   1. To resolve the value of $b here, we find its most recent declaration preceding this variable on line 2.
   2. The declaration on line 2 setting the value of $b has a right-hand side expression {$a}, which contains a variable $a.
   3. To resolve the value of $a here, we find its most recent declaration preceding this variable on line 1.
   4. The declaration on line 1 setting the value of $a has a right-hand side expression {42}, which contains the literal 42.
   5. The value of this $a resolves to "42", and so the value of this $b also resolves to "42".

2. Next, we need to first resolve the expression {$c}, which contains a variable $c.

   1. To resolve the value of $c here, we find its most recent declaration preceding this variable on line 4.
   2. The declaration on line 4 setting the value of $c has a right-hand side expression {$a}, which contains a variable $a.
   3. To resolve the value of $a here, we find its most recent declaration preceding this variable on line 3.
   4. The declaration on line 3 setting the value of $a has a right-hand side expression {13}, which contains the literal 13.
   5. The value of this $a resolves to "13", and so the value of this $c also resolves to "13".

Put together, the message resolves and formats the same as if it had been:

{The values are {42} and {13} !}

@mihnita, with reference to the above, would you be willing to acknowledge that the behaviour in these cases is explained, and does not include any circular references?

[aphillips]

@eemeli replied overnight, while I had the below to sleep on... but I think it's still worth sharing.

---

There are several reasons why a variable might want to be mutable. In fact, it might be useful not to think of them as "variables" so much as them being the "argument list" or "keys". I'll use the word variable below, but keep in mind that we are not a programming language. If anything we are closer to being a declarative language.

One reason to allow mutation is: the variable name may be referenced by the message pattern already. When the pattern is a simple message, altering the name of the variable in successive let statements and then in the expression is perhaps trivial. But when there are selectors there can be multiple expressions that reference the variable. Each such reference is probably a segment in the translation memory. Changing the name globally in the message is error prone and breaks TM leveraging. For example, here's the English (other languages will also have few, many, two, etc.):

let $foo = {$foo :transform}
match {$a :plural} {$b :plural}
when 0   0   {...{$foo}...}
when 0   one {...{$foo}...}
when 0   *   {...{$foo}...}
when one 0   {...{$foo}...}
when one one {...{$foo}...}
when one *   {...{$foo}...}
when *   0   {...{$foo}...}
when *   one {...{$foo}...}
when *   *   {...{$foo}...}

@stasm also called out that it makes for a bit of weirdness, in that externally defined variables might be overridden, but locally defined ones produce an error. If we make variables immutable, I would probably favor making it an error to override an externally supplied value (which might produce hard-to-debug errors if the argument list is dynamically generated or used across multiple messages--the latter of these is more common than one might think)

Also, we don't allow function nesting, so any transformation that requires multiple function calls results in multiple "waste values" since, as noted above, can't just "fix up" the original value. A reasonable use for reassignment is decoration or normalization of the original value. Stuff like:

let $foo = {$foo :text-transform transform=uppercase}
let $foo = {$foo :trim}
let $foo = {$foo :sanitize target=html}

... and I might very much want to do such processing in my message because (a) laboriously writing transforms into my expressions is messy and (b) if I need to nest transforms I have to write a custom function to do it (because no nesting). Yes, I can assign a new variable to each transform, but why do I have to?

I don't think anyone is proposing "forward definitions" or circular dependencies. If the message refers to an as-yet undefined value, that's an error.

One of @mihnita's questions was the answer to this riddle:

In other words, if I do let a = {42} let b = {$a} let a = {13} let c = {$a} {The values are {b} and {c} !}
What is the result?

I think it makes sense if the answer is:

And it if is "The values are 42 and 13 !", so the first definition is not "fully shadowed".

... because as noted at the start, we are not a programming language and these are not variables: they are keyed values. So $b was assigned the value of $a, not the reference to $a

The above arguments work best on primitives (string, number, etc.) and less well with values that are objects (person, currency value, inventory item, "Object Foo"), where each let assignment could require a deep copy of values on the right hand side.

I'll be interested to see how we land on this issue, as there is merit to making keyed values into constants. But I think there is reasonable utility to allowing reassignment of a key because I value utility and ease for message authors more than the internal troubles for MF implementers in this case.

[mihnita]

And many localization tools can leverage changes in placeholders already.
They leverage (and change the placeholder "on the fly") from "Hello %s" to "Hello %12s" to "Hello {0}"

But that's not the main point. We can just use another names.

the variable name may be referenced by the message pattern already

Then we can always add another intermediate name before.

For the first example:

let $foo = {$foo :transform}
match {$a :plural} {$b :plural}
  when 0   0   {...{$foo}...}
  ...
  when *   *   {...{$foo}...}

can be refactored without changing the messages proper (affected by leveraging):

let $foo_tmp = {$foo :transform1}
let $foo = {$foo_tmp :transform2}
match {$a :plural} {$b :plural}
  when 0   0   {...{$foo}...}
  ...
  when *   *   {...{$foo}...}

This one:

let $foo = {$foo :text-transform transform=uppercase}
let $foo = {$foo :trim}
let $foo = {$foo :sanitize target=html}
```
can similarly use different names, and it will not affect leveraging:

let $foo_tmp1 = {$foo :text-transform transform=uppercase}
let $foo_tmp2 = {$foo_tmp1 :trim}
let $foo = {$foo_tmp2 :sanitize target=html}

---

> so any transformation that requires multiple function calls results in multiple "waste values" since, as noted above
> ...
> And it if is "The values are 42 and 13 !", so the first definition is not "fully shadowed".
> ... because as noted at the start, we are not a programming language and these are not variables: they are keyed values.

I thing that between these 2 clarifications what we are "wasting" are not values, but value names.

If the "shadowed definition" is still accessible somehow (to give us 42 in the example) it means the whole expression still exists, without a name. So that is maybe just a minor optimization? Not the concern of a spec?

---

> because as noted at the start, we are not a programming language and these are not variables: they are keyed values.

I don't think that is obvious, or even 100% agreed.

Or, if it is for (some of) us, it is not clear for programmers, using programming languages all day long.
This looks like a programming language, but it does not behave like one.
And they use (this feature) rarely enough to not know or understand this is not the same.
The fact that we call them variables, but are not, does not help.

The example above is unclear unless you really read and really understand the spec (if already captured, and I am not sure).
For me the behavior was not clear.

Using different names makes 100% obvious without reading the spec.

---

Should I add here my comments from https://github.com/unicode-org/message-format-wg/pull/381?

> I don't think anyone is proposing "forward definitions" or circular dependencies.
> If the message refers to an as-yet undefined value, that's an error.

If the answer to my previous question was "13 ... 13", then it was a technically a circular dependency.

[mihnita]

Eemeli quotes the proposal to the spec:

If more than one declaration binds a value to the same name

I think that the meaning of "binds" is not defined.

Zibi used "binding" all the time, and I asked many times for a clarification of what that means.
And I also pointed out to existing "binding" techniques that are incompatible with the behavior he described.

TLDR: I don't think there is one universal definition of "bind" that I don't understand, and everyone else does.
So we can't just use that in a spec and assume it is understood.

There are difference between "static binding", "dynamic binding", "early binding", "late binding"

So we don't know what "bind" means here.

And we can completely bypass the need to define and understand it by just saying that the local "things" we declare & define (in one single step) are immutable.
And maybe we should not call them variables :-)

And I don't think that if we document what binding means in our context helps.
We force developers to read and understand that.
And if our meaning of "binding" is different from what they use day to day in their programming languages, we introduce "cognitive friction".

[mihnita]

A good read: https://courses.cs.vt.edu/~cs3304/Spring02/lectures/lect06.pdf