Add support for RTL languages #784
Comments
|
https://terminal-wg.pages.freedesktop.org/bidi/ has some excellent notes on how to model bidi in terminal emulators. To make progress, I need to better understand:
|
|
Can pango be an option? Although it has some gtk dependencies, it uses harfbuzz and fribidi. That's probably how gnome-terminal supports RTL. For testing, let me know if I can help by providing text content. Also, if you are using harfbuzz, shouldn't Arabic script characters get combined? Currently they don't. |
|
I forgot to say that (I think) @behdad (creator of harfbuzz) is responsive, if you have questions. |
|
There is also servo's unicode-bidi. Mentioned in alacritty/alacritty#663. |
|
There's discussion on kas-gui/kas-text#20 about bidi implementations for Rust. My impression right now is that the state of bidi in Rust is young and that the easiest path will result in a relatively slow bidi implementation, which isn't ideal: shaping already costs perf in wezterm today. Putting in more work on the promising alternative mentioned in that thread will likely be a better end-state, but will take more effort and that shouldn't be owned by wezterm. The main constraint I have right now is time: if someone has time and wants to drive this forward, I'm very receptive to seeing wezterm support bidi and helping that person figure out how to integrate it into wezterm. |
We don't do anything with these; this is just teaching the parser how to recognize these codes. refs: #784
In order to support RTL/BIDI, wezterm needs a bidi implementation. I don't think a well-conforming rust implementation exists today; what I found were implementations that didn't pass 100% of the conformance tests. So I decided to port "bidiref", the reference implementation of the UBA described in http://unicode.org/reports/tr9/ to Rust. This implementation focuses on conformance: no special measures have been taken to optimize it so far, with my focus having been to ensure that all of the approx 780,000 test cases in the unicode data for unicode 14 pass. Having the tests passing 100% allows for making performance improvements with confidence in the future. The API isn't completely designed/fully baked. Until I get to hooking it up to wezterm's shaper, I'm not 100% sure exactly what I'll need. There's a good discussion on API in open-i18n/rust-unic#273 that suggests omitting "legacy" operations such as reordering. I suspect that wezterm may actually need that function to support monospace text layout in some terminal scenarios, but regardless: reordering is part of the conformance test suite so it remains a part of the API. That said: the API does model the major operations as separate phases, so you should be able to pay for just what you use: * Resolving the embedding levels from a paragraph * Returning paragraph runs of those levels (and their directions) * Returning the whitespace-level-reset runs for a line-slice within the paragraph * Returning the reordered indices + levels for a line-slice within the paragraph. refs: #784 refs: kas-gui/kas-text#20
This commit is larger than it appears to due fanout from threading through bidi parameters. The main changes are: * When clustering cells, add an additional phase to resolve embedding levels and further sub-divide a cluster based on the resolved bidi runs; this is where we get the direction for a run and this needs to be passed through to the shaper. * When doing bidi, the forced cluster boundary hack that we use to de-ligature when cursoring through text needs to be disabled, otherwise the cursor appears to push/rotate the text in that cluster when moving through it! We'll need to find a different way to handle shading the cursor that eliminates the original cursor/ligature/black issue. * In the shaper, the logic for coalescing unresolved runs for font fallback assumed LTR and needed to be adjusted to cluster RTL. That meant also computing a little index of codepoint lengths. * Added `experimental_bidi` boolean option that defaults to false. When enabled, it activates the bidi processing phase in clustering with a strong hint that the paragraph is LTR. This implementation is incomplete and/or wrong for a number of cases: * The config option should probably allow specifying the paragraph direction hint to use by default. * https://terminal-wg.pages.freedesktop.org/bidi/recommendation/paragraphs.html recommends that bidi be applied to logical lines, not physical lines (or really: ranges within physical lines) that we're doing at the moment * The paragraph direction hint should be overridden by cell attributes and other escapes; see 85a6b17 and probably others. However, as of this commit, if you `experimental_bidi=true` then ``` echo This is RTL -> عربي فارسی bidi ``` (that text was sourced from: microsoft/terminal#538 (comment)) then wezterm will display the text in the same order as the text renders in Chrome for that github comment. ``` ; ./target/debug/wezterm --config experimental_bidi=false ls-fonts --text "عربي فارسی ->" LeftToRight 0 ع \u{639} x_adv=8 glyph=300 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 2 ر \u{631} x_adv=3.78125 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 4 ب \u{628} x_adv=4 glyph=244 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 6 ي \u{64a} x_adv=4 glyph=363 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 8 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 9 ف \u{641} x_adv=11 glyph=328 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 11 ا \u{627} x_adv=4 glyph=240 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 13 ر \u{631} x_adv=3.78125 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 15 س \u{633} x_adv=10 glyph=278 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 17 ی \u{6cc} x_adv=4 glyph=664 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 19 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 20 - \u{2d} x_adv=8 glyph=276 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 21 > \u{3e} x_adv=8 glyph=338 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs ``` ``` ; ./target/debug/wezterm --config experimental_bidi=true ls-fonts --text "عربي فارسی ->" RightToLeft 17 ی \u{6cc} x_adv=9 glyph=906 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 15 س \u{633} x_adv=10 glyph=277 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 13 ر \u{631} x_adv=4.78125 glyph=272 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 11 ا \u{627} x_adv=4 glyph=241 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 9 ف \u{641} x_adv=5 glyph=329 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 8 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 6 ي \u{64a} x_adv=9 glyph=904 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 4 ب \u{628} x_adv=4 glyph=243 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 2 ر \u{631} x_adv=5 glyph=273 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText 0 ع \u{639} x_adv=6 glyph=301 wezterm.font(".Geeza Pro Interface", {weight="Regular", stretch="Normal", italic=false}) /System/Library/Fonts/GeezaPro.ttc index=2 variation=0, CoreText LeftToRight 0 \u{20} x_adv=8 glyph=2 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 1 - \u{2d} x_adv=8 glyph=480 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs 2 > \u{3e} x_adv=8 glyph=470 wezterm.font("Operator Mono SSm Lig", {weight="DemiLight", stretch="Normal", italic=false}) /Users/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs ; ``` refs: #784
|
I've pushed a commit with what is probably the bare minimum level of support: I'm sure it's wrong in a number of cases, but with this as my test case (borrowed from microsoft/terminal#538 (comment)) Starting wezterm like this to start with the default config, then make the font bigger and turn on bidi mode: that's equivalent to running with this config: return {
font_size = 36,
initial_rows = 5,
initial_cols = 30,
experimental_bidi = true, -- this is the bit you want to use to try this out
}Pasting:
TODO:
|
|
Note that the terminal-wg bidi document, while giving the impression of being well researched, makes no mention of the DEC RTL sequences from the VT5xx terminals (e.g. DECRLM) and related modes supported by Hebrew terminal emulators like Hterm. IMO those existing modes were much more useful for anyone doing serious RTL development than any of the modern proposals. |
|
Thanks James; I'll queue up some more reading/research! |
|
@behdad I don't mean to pounce, but I wonder if you have suggestions specifically on handling the narrower glyphs in This is how that same sequence renders in Terminal.app: Even if I use the same font (which I think is the SF Arabic font), I still have gaps in my presentation. It feels like something in Terminal.app knows to stretch those ligatures and I wonder if harfbuzz has some way to express that? Or is this just deep magic in Apple's shaper/typography implementation? (Maybe sort of related: #1333 is a feature request for Devanagari support, which also has some challenging glyph widths for a terminal. Would love to hear your thoughts on that as well!) I'd also love to hear if you have other recommendations on bidi/rtl support in the context of a terminal? |
|
Currently I can report 2 issues:
The other one you mentioned yourself, a space between glyphs that are combined together; I see this problem in VTE based terminal as well, if a non-monospace font is used. If I use a monospace(DejaVu Sans Mono) font it shows correctly(in wezterm and VTE based terminal). |
Yes.
HarfBuzz doesn't know that. I haven't checked Terminal.app. It might be a geometric stretch. You sure it's using the same font? |
Looks obviously a different font. |
|
I didn't find exactly the font that Terminal.app is using, but I found that updating my local copy of Cascadia Code and using that looked better: I'll stop chasing that particular dragon :) |
Could you run:
I haven't done anything about cursor positioning or input so far. I don't know how to type this script into the terminal; could you run through how you do that? I'm assuming that you have a particular keyboard/IME configured. Could you walk me through typing a short bit of text (a couple of letters/glyphs) that mixes LTR and RTL so that I can try this for myself and not produce nonsense?
I think part of the docs to write up around this will be to suggest a good monospace font. |
Only letters(متن=text, فارسی=Persian=Farsi), which works fine:
|
|
There is Vazir Code fonts, the |
|
Thanks for this: it gives me something to play with and reason about! |
|
Thank you for working on this. |
|
@behdad At the moment, I use the UBA to produce runs of the various embedding levels (to determine the direction) and feed each of those to harfbuzz without any bidi reordering. https://harfbuzz.github.io/what-harfbuzz-doesnt-do.html doesn't explicitly say which parts of the bidi algorithm should be applied pre/post shaping. Do you have recommendations about this? I'm trying to figure out what I'm doing wrong for this example; the first grouping results in the space being reordered to the left and the last grouping has it reordered to the right. When wezterm renders these, it will render them starting from x=0 in the order they are listed below, incrementing x by the x_advance. The result is that there is no space between these runs, only around the edges. |
|
I'm far removed from bidi algorithm right now to know what the expected output is. |
The Arabic text in that screenshot is set in Courier New. |
You need to reorder the runs, but without reversion the characters in RTL runs. |
Two problems: * Need reordered_runs method to populate ranges based on the reordered levels! * Use reordered runs to get the *logical* bounds of those runs and pass those to harfbuzz. Now the text is ordered correctly, but the rendering advances by the wrong amount for the reordered clusters and looks bad unless experimental_pixel_positioning=true. refs: #784
We were using the physical cell position to place the glyphs, but we need to use the visual cell position (post-RTL-reordering). refs: #784
|
Hebrew looks great BTW, Conjoined RTLed alphabets are more complicated. |
|
Rendering RTL languages is pretty nice right now with Thanks a lot for you work, dealing with bidi stuff must be a headache! |
|
To be clearer, here is an attached image of how mlterm does it: For those wondering why this matters, consider the following scenario of typing some words and pay attention to the order of how we typed the words: Scenario 1
Scenario 2
If you look here, you will notice that wezterm renders these two lines in the same exact way, although they were typed in different order. First line is correct as it starts with English, first words inserted into nano in this case. Second line is wrong, as it should start with Arabic words first as they were typed before the English ones in this case. If the second line begins from the right side (as the case in mlterm above) this issue would be fixed. |
|
cosmic-term is using cosmic-text(which uses the rustybuzz, swash and unicode-bidi crates, ATM, I think) for its text shaping, rendering and RTL and bidirectional rendering support. I don't know the details or how good it is, but thought it wouldn't hurt to mention it. |
|
With large language models able to parse 20+ human languages, I think the support is becoming more important than ever before. I read the thread and I couldn't really understand the solutions supported so far. I tried |
|
@thisismygitrepo try |
|
I try
But there are still some problems like doesn't work in vim/neovim: |
|
Yes it worked thank you. There is only one more problem with the ZERO WIDTH NON-JOINER character with Unicode But in In And in I tried with every font and the problem was still there. I don't think that the problem is exactly with the programs themselves, such as It works fine in And in |
|
@sajadspeed See vim/vim#7932. You can conceal it in Vim. |
|
I found a bug when editing big files containing Persian text using Vim. The first time you encounter RTL text by scrolling down, it's still LTR until you press Ctrl+L. |
You should probably specify if you're talking about shaping or alignment. This may help developers identify the problem. |
|
after some scrolling, this would be the formatting : |
|
Other than that, the cursor position is so confusing in Neovim—it shows you visually somewhere, but it's physically elsewhere. |
This is a Vim issue. Unrelated to the terminal. See: vim/vim#14115 |
|
Vim doesn't support Bidi and relies upon the terminal for this behavior. In the Gnome terminal, it is somewhat better—at least it shows the cursor where it really is. |
|
There's a very good document about RTL challenges in terms of terminal display. |
|
do I have to use the same font for both LTR and RTL langauges? can I have it be different based on the input language chosen, if I switch between the two ? |
|
I don't know if this is related, but sometimes when I'm using arabic (I'm using the Kawkab Mono font) the shaping is incorrect. Neovim default shaping does some parts correctly, but gets others wrong, and wezterm default shaping does some parts correctly but gets others wrong. For example 
the 'س' and 'ل' are not connected like they're supposed to be. But if I used neovim 
The two characters are connected. I suspect this is because Neovim treats the 'ل' and 'ا' as one character instead of two. I think these are called ligatures. How would I emulate the same behavior in wezterm? |
|
I'd love to put in the time and effort to help develop bidi support in Wezterm. I have very basic familiarity with Rust and Unicode. |
|
@ProfessorMOB I'd welcome your contribution! I will note that fonts and shaping are quite complex in general and that dealing with monospace in a terminal environment is not generally well defined in the various specs, so any work done here is going to have a steep learning curve. I have fairly limited bandwidth at the moment so I'll be limited in the amount of on-boarding assistance I can provide--it's going to feel tough to get to the point where you feel like you know what you're doing here! |
|
If the adventure will be anything like my Neovim experience, I won't be climbing up the learning curve, I'm gonna attach myself to a rocket, go to space, and then ecstatically crash land on the other side of the curve. Yeah, I'm love this experience. |
|
@ProfessorMOB That has a LAM-ALEF ligature, right? That's problematic, since LAM + ALEF as separate characters would take 2 grid cells, while as a ligature would take 1 grid cell. ISTR that vim has some support for LAM-ALEF ligatures when There is a recent proposal by mintty to add an escape sequence to enable/disable LAM-ALEF ligatures in terminal emulators: https://gitlab.freedesktop.org/terminal-wg/bidi/-/issues/4#note_2744194 |
|
@ninjalj yes! |
|
@ProfessorMOB when doing Arabic shaping, vim/neovim transform arabic characters into their positional forms in the Unicode Arabic Presentation Forms B block, so maybe the difference comes from there? You could try |
|
@ninjalj Bookmarked! I feel part of the problem is that because it uses that block, it doesn't render all the arabic letters correctly. Is it possible to reuse part of the block but then use another part of what Wezterm uses by default to get the best of both worlds? |
The presentation forms blocks are meant only for interfacing with legacy sw/hw, in this case, for interfacing with terminal/terminal emulators that don't do shaping. So, ideally, on a terminal which does shaping, like wezterm, you could do |
|
@ninjalj So this issue is a wezterm bug? Is wezterm supposed to know how to shape the lam-alef ligature? Also, what bindings (if any) does wezterm use in order to work with Harfbuzz? Harfbuzz is a C library, but Wezterm is written in Rust. |
|
@ninjalj On another note as well, should I raise this as a separate issue on wezterm? |
Is your feature request related to a problem? Please describe.
wezterm cannot display right to left languages correctly. RTL text is not not RTL, and characters that need to be combined, are not.
Describe the solution you'd like
Support RTL text. Probably needs bidirectional text handling and text shaping.
Related projects: harfbuzz, FriBidi
Describe alternatives you've considered
Konsole and gnome-terminal support RTL languages.
Additional context
Image, although the image is comparing Konsole with Alacritty, wezterm works just like Alacritty.
The text was updated successfully, but these errors were encountered: