Block comment before regex literal with quote can break highlighting

[nwhetsell]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

Here’s an example with a block quote before a regex literal:

var test = /* */ /"/;
// comment
// comment
// comment

Here’s an example without the block quote:

var test =       /"/;
// comment
// comment
// comment

[iskitz]

I've seen and continue to be able to reproduce this issue by using an unmatched single or double quote character within a regex. Here're some examples:

;
+ /this one's ok/
- /and this should've been ok/
* /but it's not :-(/

~ /this one's also not ok/
+ /this one's also not though using a single " is uncommon/
;

If this scenario seems contrived please see ionify's readme, for a project that makes frequent use of this type of regex syntax.

I've added to this issue because
1: I ran into this via GitHub's syntax highlighting
2: GitHub claims to use atom/language-javascript & requests filing issues here rather than at github/linguist

[winstliu]

The problem with this issue is that language-javascript can't tell the difference between 3 /* */ / 4 and regex = /* */ /'/. It's quite tricky.

[iskitz]

Thanks @50Wliu. Can you explain a bit more why that's so?

I took a quick look at this repo's code but would need to spend more time than I currently have to interpret its algorithm & decipher why this is such a difficult problem. Prior to looking, I thought it might've been an issue with the order in which tokens are recognized but I couldn't immediately or quickly confirm that.

A current, not so pretty, workaround I've found is to wrap the problematic regexes in parenthesis like:

;
~(/ This style isn't problematic but needs 2 more characters /)
;

[winstliu]

Sure. The fundamental issue is that division and regex share the same character /. Therefore when we hit a / language-javascript needs to figure out whether it's referring to division or the start of a regex.
The Oniguruma regex engine only allows fixed-width lookbehinds, meaning that we're limited to very basic regexes when looking at what came before the / for context. Since division occurs more frequently than regex, we've put the lookbehind on the regex pattern.

A block comment before the regex is impossible to match using a fixed-width regex (because we have no idea how long it is) and does not give any helpful context, hence this issue.

I'll take a look at the cases you posted and see if I can fix some of those.

[winstliu]

Ok, I believe your cases are fixable (though not exactly what one would call valid JavaScript...) - I'll create a PR shortly.

[iskitz]

Thanks for the explanation & PR #542 @50Wliu!

Glad to hear that prefixed punctuation's easier to fix. Although the samples I provided don't look typical, they're all valid JavaScript 😉 You can read more about that here.

question: Would the regex you changed need to be updated if there're more punctations that could be used?

I ask because the full set of ionify's punctuation can be any of the +, -, *, /, or % arithmetic operators, &, ~, |, ^, <<, or >> bitwise operators, or >, >=, <, or <= relational operators.