Document considerations related to lowering the collation strength from the default #6662
Conversation
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback. |
|
| @@ -15,13 +15,45 @@ use crate::{ | |||
| CollatorPreferences, | |||
| }; | |||
|
|
|||
| /// The collation strength that indicates how many levels to compare. | |||
| /// The collation strength that indicates how many levels to compare. The primary | |||
| /// level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é' | |||
There was a problem hiding this comment.
Is it guaranteed to be the "base letter"? How does that map to other writing systems, or to locales that sort certain diacritics nonadjacent to their base letter (like Danish Å)?
There was a problem hiding this comment.
On the one hand: I don't think it's guaranteed to be a base letter, per se. For example, the Rupee sign U+20A8 used to sort with other currency symbols. Now it sorts with the ASCII letter sequence "Rs". The German sharp S (ß) sorts with ss, I believe.
On the other hand: the ICU User Guide uses the terms "base letter" and "base character": https://unicode-org.github.io/icu/userguide/collation/concepts.html
I think we can reuse ICU terminology for ICU4X docs, at least here for the purposes of illustrating the effects of different strength levels to explain strength.
| /// compare as equal. This may make sense when sorting more complex structures | ||
| /// where the string to be compared is just one field, and ties between strings | ||
| /// that differ only in case, accent, or similar are resolved by comparing some | ||
| /// secondary field in the larger structure to be sorted. |
There was a problem hiding this comment.
Does this use case actually make sense? You say in the next paragraph that it doesn't really make sense.
There was a problem hiding this comment.
I think the subsequent paragraph is just expounding on the main point of this paragraph. If so, then I think that the subsequent beginning with the word "However" is confusing for the reader.
| @@ -15,13 +15,45 @@ use crate::{ | |||
| CollatorPreferences, | |||
| }; | |||
|
|
|||
| /// The collation strength that indicates how many levels to compare. | |||
| /// The collation strength that indicates how many levels to compare. The primary | |||
| /// level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é' | |||
There was a problem hiding this comment.
On the one hand: I don't think it's guaranteed to be a base letter, per se. For example, the Rupee sign U+20A8 used to sort with other currency symbols. Now it sorts with the ASCII letter sequence "Rs". The German sharp S (ß) sorts with ss, I believe.
On the other hand: the ICU User Guide uses the terms "base letter" and "base character": https://unicode-org.github.io/icu/userguide/collation/concepts.html
I think we can reuse ICU terminology for ICU4X docs, at least here for the purposes of illustrating the effects of different strength levels to explain strength.
| //! The degree of sensitivity in how to determine that strings are distinct. | ||
| //! The collation strength indicates how many levels to compare. The primary | ||
| //! level considers base letters, i.e. 'a' and 'b' are unequal but 'E' and 'é' | ||
| //! are equal, with further levels dealing with distinctions such as accents |
There was a problem hiding this comment.
Just to clarify that the "high" / "low" relationship corresponds to primary = lowest, indentical = highest
| //! are equal, with further levels dealing with distinctions such as accents | |
| //! are equal, with higher levels dealing with distinctions such as accents |
| //! If an earlier level isn't equal, the earlier level is decisive. | ||
| //! If the result is equal on a level, but the strength is higher, | ||
| //! the comparison proceeds to the next level. |
There was a problem hiding this comment.
| //! If an earlier level isn't equal, the earlier level is decisive. | |
| //! If the result is equal on a level, but the strength is higher, | |
| //! the comparison proceeds to the next level. | |
| //! If an lower level isn't equal, the lower level is decisive. | |
| //! If the comparison result is equal on one level, | |
| //! but the collator's strength input value is higher than that, | |
| //! then the collator comparison iteratively proceeds to the next higher level. |
| //! Note that lowering the strength means that more user-perceptible differences | ||
| //! compare as equal. This may make sense when sorting more complex structures |
There was a problem hiding this comment.
| //! Note that lowering the strength means that more user-perceptible differences | |
| //! compare as equal. This may make sense when sorting more complex structures | |
| //! Note that lowering the strength value given to the collator means that more user-perceptible | |
| //! differences will compare as equal. This may make sense when sorting more complex structures |
| //! compare as equal. This may make sense when sorting more complex structures | ||
| //! where the string to be compared is just one field, and ties between strings | ||
| //! that differ only in case, accent, or similar are resolved by comparing some | ||
| //! secondary field in the larger structure to be sorted. |
There was a problem hiding this comment.
optional: you could mention that some browsers implement the "Find" functionality for a page based on collation, ex: via the StringSearch API in ICU. This allows them to choose between primary strength so that it can ignore diacritics and case ("koln" matching on "Köln") or secondary strength to only match exactly on diacritics ("Bár" not matching "Bär").
| //! that differ only in case, accent, or similar are resolved by comparing some | ||
| //! secondary field in the larger structure to be sorted. | ||
| //! | ||
| //! However, if the sort is just a string sort without some other field for |
There was a problem hiding this comment.
In general for this paragraph, I think it should be worded as a detailed caution to the user. It expounds on the main point of the previous paragraph, IMO.
| //! However, if the sort is just a string sort without some other field for | |
| //! Therefore, if the sort is just a string sort without some other field for |
| //! provide the stability property) affect the relative order of strings that | ||
| //! do have user-perceptible differences particularly in accents or case. | ||
| //! | ||
| //! Lowering the strength is less of a perfomance optimization that it may seem |
There was a problem hiding this comment.
| //! Lowering the strength is less of a perfomance optimization that it may seem | |
| //! Lowering the strength is less of a perfomance optimization than it may seem |
| //! Lowering the strength is less of a perfomance optimization that it may seem | ||
| //! directly from the above description. As described above, in the case | ||
| //! of identical strings to be compared, the algorithm has to work though all | ||
| //! the levels included in the strength without an early exit. However, this |
There was a problem hiding this comment.
| //! the levels included in the strength without an early exit. However, this | |
| //! the levels, from primary up to the provided strength value given to collator, without an early exit. However, this |
| /// compare as equal. This may make sense when sorting more complex structures | ||
| /// where the string to be compared is just one field, and ties between strings | ||
| /// that differ only in case, accent, or similar are resolved by comparing some | ||
| /// secondary field in the larger structure to be sorted. |
There was a problem hiding this comment.
I think the subsequent paragraph is just expounding on the main point of this paragraph. If so, then I think that the subsequent beginning with the word "However" is confusing for the reader.
Inspired by discovering how Firefox and Thunderbird UI code uses collation and pondering whether it makes sense.