Design decisions re selectable newline character

[trespaul]

I've been using Helix for several years now, and it took me a while to get used to there being an extra "character" at the end of the line that will mess things up if I accidentally include it in a selection. But I was recently reminded of its strangeness when I got annoyed by an extra blank line in the middle of a paragraph that gets rendered when the last text character lands on the last column of the editor width, and I want to start the next line on a newline without adding a paragraph break (two newlines). The selectable newline character is pushed onto the next line, which makes it look as if I deliberately put an extra newline there, and makes it hard to tell paragraphs apart. The only way to tell this is not a paragraph break is by toggling visible whitespace (which adds unwanted visual clutter) or by noticing that there is no line number on that line. (I think this is unexpected behaviour not yet mentioned in issues like #3076 and #2956.)

(This specific issue could also be solved by having soft-break break on a previous whitespace character if the only character(s) on the new line would be the newline character, or only a period or semicolon, etc., i.e., keep the last W-word on a line together.)

So, this has come up a few times before from what I could find, but what I couldn't find was a comprehensive discussion of the thought behind the choice (I didn't search the matrix room though). The most direct discussion I could find was #8365 which links #362 but unless I'm misunderstanding something, there doesn't seem to be anything there that prevents hiding the selectable newline. Things have also changed in the intervening years (e.g., x doesn't work like desscribed in the Example 2 there).

I'm trying to think what the possible (high-level) reasons could be to have a selectable newline.

Firstly, one could argue that, well, the newline is a unit in the text file that is being represented. But if we think of a text file as a string of units, we render the file by doing something specific for each unit, the units are not rendered out verbatim. So, e.g., combining codepoints are ... combined. Now, in my mind, it makes sense that when a newline unit appears, one should follow the "instruction" and render what we generally consider to be what a "newline character" indicates, i.e., a new line. A selectable character representing the newline would then be in addition to the new line already rendered. If one wanted to be consistent, with minimal "edge-case logic", then one should only render one representation of the source instruction. If one wants a separate selectable character at the end of a line that represents the newline, then one should not also render a line break.

The other justification I could think of is that it is there for functional reasons. Having the newline character selectable enables one to do actions that would not otherwise be possible. But this is not true, since there are already line-specific operations separate from character-operations. E.g., to delete a new-line "character", one can combine the lines with J (which collapses whitespace but that's besides the point). Everything else I could think of that one might do with a selectable newline could also be achieved by other means. Does anyone perhaps have an example of operating on the newline character that they use regularly?

Something else that I've seen mentioned is that having a selectable newline character leads to more consistent behaviour with various commands, and that the project should try to not have many edge-cases where logic suddenly changes. However, I noticed that, e.g., xc already breaks consistency. If I selected a line, including the newline at the end of the line, and I delete the line, I expect the rendered line to disappear, i.e., no extra newline is left behind. But when I do c instead of d, suddenly an extra newline is seemingly inserted. So, c does edge case logic that make it so that it is not equivalent to di. I tried to find any reference to this as a decision about the design / user interfacing philosophy, but I could not find any. I found the PR that seemed to introduce this behaviour, and it didn't mention this inconsistency it creates. It was included with the feature of auto-indenting after pressing c (#7316). Now, this makes sense to me, it's a good feature to have, but it does show that "consistency" is already broken.

So if it's not supported by design philosophy, for feature reasons, or for consistency, then why still hold on to the selectable newline? Especially if it confuses newcomers and frustrates others by causing unexpected behaviour.

I must then assume that there are implementation details that lead to this, and that these details would make it complicated to change behaviour. But even then it could still be decided on and added to a backlog to-do list somewhere, right?

What am I missing? Or, at least, can someone please point me towards somewhere where an explanation was given? I genuinely could not find one. Each discussion seems to end with something like

This is something we rejected repeatedly and orthogonal to that option. It's an intentional part of helix editing model that the last line is selectable.

(from the first comment on #8365).

Now, I get that it must be frustrating that it comes up again and again, and it's clear that the person quoted is exasperated. But if it comes up repeatedly, it means that multiple people are wondering about it, so, if the behaviour will definitely never change, it seems worth it to properly introduce it and justify why it is the way it is. I would even say that such an explanation, if this behaviour is this core to the Helix model — and if more than a handful of people are confused by it — should be added to the documentation and/or the tutorial. If I can develop my understanding of why things work this way, I'm happy to write such an explanation for the docs myself.

---

(I'm always concerned when starting a discussion in which I disagree with the way things are done that I come across as antagonistic, so I just want to caveat that I really appreciate everyone's efforts on Helix, and that it has been a breath of fresh air to use. My disagreement about the selectable newline character is minor and not a barrier whatsoever. Finding an explanation without any changes implemented on the project would be an entirely satisfactory end to this discussion for me.)