Semicolon handling at top level is not quite right

[DavisVaughan]

I think that at top level semicolons have to be preceded by an expression?

Interestingly, inside a { any number of semicolons are blindly consumed

It's possible that semicolons are only valid:

• At top level, when preceded by an expression
• Directly inside a { scope (but not nested inside one)

I also wonder if we should emit semicolons in the parse tree as real tokens. Otherwise it just looks like whitespace to downstream consumers.

# Top level `;` is a parse error when its all alone, but we happily consume it
parse(text = ";")
#> Error in parse(text = ";"): <text>:1:1: unexpected ';'
#> 1: ;
#>     ^
treesitter::text_parse(";", treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> ;
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 1)])

# We eat them all, when we should error
treesitter::text_parse(";;;", treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> ;;;
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 3)])

# This is fine at top level
parse(text = "1;")
#> expression(1)
treesitter::text_parse("1;", treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> 1;
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 2)]
#>   (float [(0, 0), (0, 1)])
#> )

# Interestingly this works (and we parse that fine)
parse(text = "{ ; }")
#> expression({ ; })
parse(text = "{ ; ; }")
#> expression({ ; ; })
treesitter::text_parse("{ ; }", treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> { ; }
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 5)]
#>   (braced_expression [(0, 0), (0, 5)]
#>     open: "{" [(0, 0), (0, 1)]
#>     close: "}" [(0, 4), (0, 5)]
#>   )
#> )
treesitter::text_parse("{ ; ; }", treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> { ; ; }
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 7)]
#>   (braced_expression [(0, 0), (0, 7)]
#>     open: "{" [(0, 0), (0, 1)]
#>     close: "}" [(0, 6), (0, 7)]
#>   )
#> )

# Note that this doesn't work
parse(text = "x[;]")
#> Error in parse(text = "x[;]"): <text>:1:3: unexpected ';'
#> 1: x[;
#>       ^

# Nor does this, so its not like with newlines where newlines are consumed
# recursively within a `(` / `[` / `[[` scope
parse(text = "{ x[;] }")
#> Error in parse(text = "{ x[;] }"): <text>:1:5: unexpected ';'
#> 1: { x[;
#>         ^

[DavisVaughan]

• At top level semicolons definitely have to be preceded by an expression
  • https://github.com/wch/r-source/blob/988774e05497bcf2cfac47bfbec59d551432e3fb/src/main/gram.y#L449
• Inside an exprlist (which is the body of a { } expression), semicolons can follow any exprlist (which itself can be empty, meaning ; can float anywhere in there, as we've seen), or be between two expressions
  • https://github.com/wch/r-source/blob/988774e05497bcf2cfac47bfbec59d551432e3fb/src/main/gram.y#L468
  • https://github.com/wch/r-source/blob/988774e05497bcf2cfac47bfbec59d551432e3fb/src/main/gram.y#L537-L540
  • Note this behavior isn't recursive, so just because you open a { scope doesn't mean a semicolon can now magically be anywhere deeply nested inside that scope. It just applies to the first level of the { scope.

Notably you can't do _;, that's a special pipe placeholder parse error
https://github.com/wch/r-source/blob/988774e05497bcf2cfac47bfbec59d551432e3fb/src/main/gram.y#L1755