for comprehension loses backticks when desugaring to val?

[jducoeur]

This arose from this thread on Scala Users.

Take the following code (working fiddle, Scala 2.12):

val `P(X|C)`=3

println(`P(X|C)`)

val x = for {
  `F(Y|Z)` <- Option(3)
  `P(X|C)` <- Option(6)
  `A(B|C)` = 4  // fails here
}
  yield `P(X|C)`
  
println(x)

Everything compiles for me except the specified line, which is getting the error:

ScalaFiddle.scala:8: error: not found: value A(B|C)
    `A(B|C)` = 4
    ^

All of this suggests to me that something's a bit weird with backtick'ed identifiers when desugaring to a val. The desugars to flatMap and map appear to work, but the one to val doesn't...

[joroKr21]

Maybe it desugars to a pattern match? Typer says:

       private[this] val x: Option[Nothing] = scala.Option.apply[Int](3).flatMap[Nothing](((F(Y|Z): Int) => Option(6).map(((x$3: Int) => x$3: @scala.unchecked match {
  case (x$1 @ `P$u0028X$barC$u0029`) => {
    val <x$2: error>: <error> = 4: @scala.unchecked match {
      case (<x$2: error> @ `<A$u0028B$barC$u0029: error>`) => <x$2: error>
    };
    scala.Tuple2(x$1, <x$2: error>)
  }
})).map(((<x$4: error>: <error>) => <x$4: error>: @scala.unchecked match {
          case scala.Tuple2(`P$u0028X$barC$u0029`, `<A$u0028B$barC$u0029: error>`) => `P$u0028X$barC$u0029`
        }))));

[szeiger]

The spec is vague on this topic. The only thing I'm sure about is that something is wrong.

Starting with regular value definitions (https://www.scala-lang.org/files/archive/spec/2.12/04-basic-declarations-and-definitions.html#value-declarations-and-definitions). The formal syntax for the LHS is always at least a Pattern2, so the "simple name" case is not distinguished at the syntactic level. It is listed explicitly in the description:

Value definitions can alternatively have a pattern as left-hand side. If p is some pattern other than a simple name or a name followed by a colon and a type, then the value definition val p = e is expanded as follows

There is no formal definition of "simple name" in the spec. According to use of the term in earlier chapters it refers to an unqualified identifier which may also be "an arbitrary string between back-quotes".

The implementation of value definitions agrees with this. Any unqualified identifier, even when enclosed in backquotes, is treated as a simple name. No pattern matching applies.

According to https://www.scala-lang.org/files/archive/spec/2.12/06-expressions.html#for-comprehensions-and-for-loops:

A generator p <- e produces bindings from an expression e which is matched in some way against pattern p. A value definition p = e binds the value name p (or several names in a pattern p) to the result of evaluating the expression e

This could be interpreted to mean that p in a generator is always a pattern but that it can be a simple name in a value definition. Or it could be an imprecise way of saying that there are patterns that bind a single name p. My guess would be that it was originally intended to be the same as a normal val (remember, back in the day the syntax required a val keyword in this case) but maybe this was changed to align it with generators when the val keyword got removed. (Spoiler alert: The current implementation disagrees with both readings.)

This brings us to https://www.scala-lang.org/files/archive/spec/2.12/08-pattern-matching.html#stable-identifier-patterns:

To resolve the syntactic overlap with a variable pattern, a stable identifier pattern may not be a simple name starting with a lower-case letter. However, it is possible to enclose such a variable name in backquotes; then it is treated as a stable identifier pattern.

It does not say anything about other identifiers in backquotes. In practice, all identifiers in backquotes are interpreted as stable identifier patterns in proper patterns (i.e. not ones that are treated as simple names).

But since generators in for comprehensions are always patterns (according to the definition above), the first line (`F(Y|Z)` <- Option(3)) should not compile and the second line (`P(X|C)` <- Option(6)) should be a stable identifier pattern. So in practice generators do recognize simple names and treat them as simple definitions, not as pattern matches. This is consistent with value definitions. However, value definitions also contain an exception for what would otherwise be a typed pattern (val (x: String) = (1: Any), just like val x: String = (1: Any), is not a pattern match) but this is not mirrored by generators.

As the original bug report in this ticket shows, value definitions in for comprehensions are not consistent with either standalone value definitions or generators. They do not make the same exceptions for simple definitions. These are always treated as proper pattern matches.

By the way, when it comes to generators, there are other oddities. Take this simple typed pattern:

for((a: String) <- Option(1: Any)) yield a

It should desugar to:

Option(1: Any).withFilter { case (a: String) => true; case _ => false }.map { case (a: String) => a }

But actually it becomes

Option(1: Any).withFilter { case (a: String) => true; case _ => false }.map { (a: String) => a }

which doesn't compile. It looks like the compiler is isn't quite sure whether it's a typed assignment or a typed pattern.

[som-snytt]

They are all patterns. It seems to me the problem is with the other generators; the back-ticked pattern is refutable.

[szeiger]

Syntactically they are all patterns, but as I elaborated (far too long) above, that's not sufficient for deciding on their interpretation. Syntactically, val x = 1 is a pattern match. (x is parsed as a Pattern2), so is val (x: Int) = 1 ((x: Int) is a Pattern2), but curiously in val x: Int = 1 only x is parsed as a Pattern2. That's why you have to look to the explanations in the spec for the interpretation of certain simple patterns.

Refutability does not matter at that point. If you see `foo` you first have to decide if that means foo (simple name) or $x if $x == foo (stable identifier pattern).

[som-snytt]

I think you're overthinking it. The syntax to introduce a funky name is:

for (`funky name` @ _ <- List(27))

which corresponds to this syntax which was enhanced in 2.12, because previously it had to be a varid:

27 match { case `funky name` @ _ => `funky name` }

Ordinary val def syntax to introduce a name is not a pattern, irrespective of the productions.

The text says the LHS can also be a pattern, etc. Of course this leads to differing expectations about what the x: String syntax does in a generator (filtering or throwing or compile error, I don't even know what it does without testing).

[martijnhoekstra]

for (`funky name` <- List(27)) Does introduce a funky name too though, which might be the bug.

…

On Fri, May 17, 2019, 18:38 som-snytt ***@***.***> wrote: I think you're overthinking it. The syntax to introduce a funky name is: for (`funky name` @ _ <- List(27)) which corresponds to this syntax which was enhanced in 2.12, because previously it had to be a varid: 27 match { case `funky name` @ _ => `funky name` } Ordinary val def syntax to introduce a name is not a pattern, irrespective of the productions. The text says the LHS can also be a pattern, etc. Of course this leads to differing expectations about what the x: String syntax does in a generator (filtering or throwing or compile error, I don't even know what it does without testing). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11533?email_source=notifications&email_token=AAGXOEM67TMTHOQNQZYLK43PV3NPNA5CNFSM4HNCRW3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVIADI#issuecomment-493518861>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXOEORVYZKQ45YCIGSPULPV3NPNANCNFSM4HNCRW3A> .

[szeiger]

I think you're overthinking it

I prefer to call this "others are underspecifying it"

[som-snytt]

@martijnhoekstra Yes, IMHO, that is the bug because it should be taken as a pattern. But I wouldn't be averse to rewriting the rules, either, or clarifying them the other way.

The umbrella issue is #900 which reminds me, I have a PR to revive that helps with retronym's workaround, to use x @ p to force "take as a pattern".

[som-snytt]

Scala 3:

11 |      `A(B|C)` = 4  // fails here
   |      ^^^^^^^^
   |      Not found: A(B|C) - did you mean P(X|C)?

Scala 2 attempts to be helpful in its own way:

Identifiers that begin with uppercase are not pattern variables but match the value in scope.
      `A(B|C)` = 4  // fails here
      ^

Note that Scala 3 is now aligned with Scala 2 in accepting

case `X` @ p =>
case `type`: Type =>

Scala 3 adds

10 |      `P(X|C)` <- Option(6)
   |      ^^^^^^^^
   |pattern's type (C.this.P(X|C) : Int) does not match the right hand side expression's type (x$1 : Int)

where Scala 2 is @unchecked (and -Xlint:valpattern does not warn because it is for val patterns, not patterns in for comprehensions).

Scala 3 clarifies the "generator" case by adding case:

scala> for (X <- List(42)) yield X
val res0: List[Int] = List(42)

scala> for (case X <- List(42)) yield X
-- [E006] Not Found Error: -----------------------------------------------------
1 |for (case X <- List(42)) yield X
  |          ^
  |          Not found: X
  |
  | longer explanation available when compiling with `-explain`
-- [E006] Not Found Error: -----------------------------------------------------
1 |for (case X <- List(42)) yield X
  |                               ^
  |                               Not found: X
  |
  | longer explanation available when compiling with `-explain`
2 errors found

similarly

scala> for((a: String) <- Option(1: Any)) yield a
-- Error: ----------------------------------------------------------------------
1 |for((a: String) <- Option(1: Any)) yield a
  |        ^^^^^^
  |pattern's type String is more specialized than the right hand side expression's type Any
  |
  |If the narrowing is intentional, this can be communicated by adding the `case` keyword before the full pattern,
  |which will result in a filtering for expression (using `withFilter`).
1 error found

scala> for (case (a: String) <- Option(1: Any)) yield a
val res0: Option[String] = None

Scala 2 is stuck at unhelpful desugaring as shown above

Welcome to Scala 2.13.16 (OpenJDK 64-Bit Server VM, Java 23.0.2).
Type in expressions for evaluation. Or try :help.

scala> for ((a: String) <- Option(1: Any)) yield a // print

scala.Option.apply[Any](((1): scala.Any)).withFilter(((check$ifrefutable$1: Any) => (check$ifrefutable$1: @scala.unchecked) match {
  case (a @ ((_): scala.Predef.String)) => true
  case _ => false
})).map[B](((a: String) => a))

(It should always rewrite to cases, which will synthesize the correct function.)

On "simple name": "simple" is an Oderskyism, as in "Simply Scala". In the recent doc for givens, a reviewer requested he revise "simple" to "mere". "If the given is a mere alias."

An example from Scala 3 where an assignment can only be construed as a pattern is

for (i <- List(42); given Int = i) yield summon[Int]

A def may say given j: Int = i, which not permitted in a pattern.