Support for continuous script languages

[xamgore]

1. Original repo doesn't provide configurable min_term_length (see this issue) and it is hardcoded as 3 here. We need it especially for continuous script languages, like Japanese, Chinese and Thai, where one symbol can represent one word.

2. Japanese language has words with symbols in the end, like コンピューター (computer), テレビー (television), ツイッター (twitter) and so on. Yake uses web_tokenizer() from segtok.tokenizer which separates ー in the end. So for continuous script languages we apply tokenization explicitly before passing text to Yake and introduce condition where tokenizer in Yake only splits words by space.

• Example solution in the LIAAD-forked repo
• Is Yake suitable for Chinese corpus？ INESCTEC/yake#87
• Does yake surpport Chinese words ？ INESCTEC/yake#40
• Will Yake support asian languages such Korean or Japanese? INESCTEC/yake#41
• Unsupported YAKE languages INESCTEC/yake#58

[bunny-therapist]

Getting rid of the custom stuff for the StopWords class and just having it be a HashSet would make this a lot easier, wouldn't it? I really think the stopword logic involves too much for us to be able to put it in a separate class. The StopWords class should just be a container for the words, not have logic for evaluating them. With that, we can have the minimum word length just be another config for Yake.

[xamgore]

“A Chinese text similarity algorithm based on Yake and neural network” 2022 paper has quite a few wise insights:

1. Chinese has no capital letters.
   • so $\mathbb{T}_{\mathrm{case}}$ feature is replaced with the word span feature
   • the feature is described below, and has no explanation, why it works
   • LIAAD computed statistical feature performance, here they don't do it
2. There is no edit distance in the way we understand it.
   • but keywords can be translated into a vector space and deduplicated by $\cos$ similarity.
3. They use LTP Chinese tool of Harbin Institute of Technology to perform Chinese word segmentation.
4. Stopwords are taken from:
   • Harbin Institute of Technology stop word thesaurus
   • Sichuan University Machine Learning Intelligent Laboratory stop word database
   • Baidu stop word list

Word span feature

This feature represents the distance between the first occurrence and the last occurrence of a word in the text, which can reflect the theme of the text. The larger the word span, the more important the word is to the text.

$\mathrm{span}(w_i) = \dfrac{ \mathrm{last}(w_i) - \mathrm{first}(w_i) + 1 } { \mathrm{wordAll} }$

Where $\mathrm{first}(w_i)$ is the position where the word $w_i$ first appears in the text. $\mathrm{last}(w_i)$ is the position where the word $w_i$ appears last in the text. $\mathrm{wordAll}$ is the total number of all words.

Experiments show their method YANN is (naturally) the best:

Method	Precision $P = TP \div ( TP + FP)$	Recall $R = TP \div ( TP + FN)$	F1 $(2 * P * R) \div ( P + R)$
TF-IDF	0.776	0.783	0.779
RAKE	0.743	0.741	0.742
TextRank	0.757	0.765	0.761
TopicRank	0.780	0.786	0.783
YANN	0.823	0.832	0.827

[bunny-therapist]

We should maybe also consider whether it is worth it to go for e.g. Chinese and Japanese if there are other keyword extractors that work better for them already.