Bidirectional structured text

[tomerm]

@aphillips @r12a @matial the use cases described in the current document (formats based on JSON, WebIDL, and other non-markup data languages) and best practices developed to provide a solution for such use cases ring the bells for me. @matial and myself have been tackling a slightly more general (a higher level if you wish) formatting problem uniquely associated with bidirectional scripts (i.e. Arabic / Hebrew). We coined a term: "structured text problem" in order to describe it.

Conceptually any kind of structure starting from simplest concatenation like Author: Mark Twain and ending with as complex as regular expression or Java code, falls in the same category of structured text as long as it includes bidirectional text.

I list here several broad categories of structured text:

Simplest structure

• Date / time stamps (4 Oct 2018)
• File path (c:\root\folder\file.txt)
• Breadcrumb (first -> second -> third)
• Concatenations (a(b), a-b, a:b ets.)

Moderate structure + a lot of political exposure

• URL
• Email address

Complex structure

• Math formulas
• Regular Expressions
• Programming language (SQL query, Java, C, C++ ...)
• Markup editor (HTML, XML etc.)

In all cases above it is not enough to just specify for specific piece of text in which direction it should be displayed. In addition:

• It is necessary to enforce the syntax ( which makes the expression readable) and which without any additional work won't be enforced in general case by UBA alone.
• It is necessary to specific and enforce display rules - in some Arabic countries math formulas run from left to right, while in others they run from right to left.

The latest document authored by @matial and myself proposing a general approach for tackling structured text problem is available from: https://docs.google.com/document/d/1y9LhT7rbGGVHjh2uqTAYHzN5PfbAkPxO5sMJygOPc3I/edit#heading=h.b43z973xff51

Do you believe it makes sense to extend the scope of your document and touch the subject of structured text ? Alternatively, would you agree to participate in a work on additional document dedicated to that subject ?

PS. Even though this specific article is not talking about specific implementation for problem solution (and thus at least in my eyes it means that this not in the scope for this article), I do wish to mention that at least for me the subject is at the very least 10 years long. During those years many solutions were implemented in different technologies (both properietary to IBM and open source such as Eclipse / ICU / Dojo etc. ).

[aphillips]

@tomerm In recent edits we have added some low-level illustration of the simple/modest structure problems. Our document's focus is on low level data structures and file formats and the need for direction metadata. Higher level protocols and implementations need this in order to deal with the problems you highlighted in this issue.

While the concept of "bidi isolation" provides the necessary "fix" for many of these issues, the fact is that UBA and plain text strings are being uses for very many of these cases and fixing them to have runtime-supplied isolating semantics (i.e. your "lot of political exposure") is complicated on a good day.

For now... are we explaining the problem(s) well enough? And are we making the right low-level recommendations, such that the larger problems might be/become solvable?