Segtok error with large amounts of text

[bchigarev]

The example given at github.com/quesurifn/yake-rus works well with a 3 MB input.txt text file. However, it gives an error with a 30 MB file.
.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\segtok-0.1.5\src\segmenter\mod.rs:197:54:
called Result::unwrap() on an Err value: RuntimeError(BacktrackLimitExceeded)

[bunny-therapist]

@xamgore An issue for segtok?

[bchigarev]

It just says this in the error: segtok-0.1.5\src\segmenter\mod.rs:197:54
I'm not a programmer, but I'm trying to use yake-rus for large texts and I get the error

[bunny-therapist]

Yes, I understand. The error is coming from segtok. @xamgore maintains that and should know what to do. (I assume either make it work for large files or document some restriction.)

[bchigarev]

Yes, that's right. There is no really working alternative to Yake! for large texts. For example, you can evaluate the topic of Chapter 1 by comparing its keywords with the keywords of the entire text. And the whole text can be large.

[xamgore]

@bchigarev, could you share the file, please?

I believe the problem is not in the file size itself, but certain symbols or words causing the bug.

BacktrackLimitExceeded error is tricky. It comes from DFA/NFA library as a safety measure for untrusted input. Without it there is a chance for a malicious user to DOS the system. How? By providing an input string which requires exponential number of steps to recognize the pattern.

The right approach would be extending Yake's Config with backtrack_limit field.

[bchigarev]

I've simplified the text. I converted it to lower case and removed all [^a-z0-9\ \.\,\?]. If I take a 3 megabyte piece of text from a 30 megabyte file, everything is fine. I took different pieces. On a large file, the error

[bchigarev]

"extending Yake's Config with backtrack_limit field". I'm not a programmer. What should the line "use yake_rust::{get_n_best, Config, StopWords};" look like in the above example for the backtrack_limit field?

[bchigarev]

I wanted to say that in "let config = Config { ngrams: 3, ..Config::default() };"

[bchigarev]

I did not find such a field in the “pub struct Config {...” structure

[xamgore]

It's just my thoughts said out loud. I still need a file to test my fix works.

For now the recommended way to overcome the problem, is stripping the input string 30 MB → 3 MB. Keywords usually lie in the first part of texts, so it should be ok.

[bchigarev]

If finding a word at the beginning of a text were more important than other criteria, this algorithm would be useless for large texts. I experimented with Yake! in Python, and there was a difference. For medium-sized texts, Yake! in Rust seemed to be faster. But I didn't save my records.
I consider Yake! on Rust to be a faster solution for large texts and a large number of keywords.

[bunny-therapist]

Yeah, yake-rust is like 10x faster than the python version and can also run multithreaded :)

@xamgore Is the problem the file size or not? Could you confirm? If this is some DDOS prevention thing, I suppose it could be affected by both file size and the characters themselves.

What would the "backtrack limit" config be? An integer or something? I think making this configurable is a good solution, but I am interested in the resulting interface being understandable to users.

[bchigarev]

I think the problem lies in the size of the text file. I use other utilities for text files that are tens of megabytes in size. For example, lemmatizer or ripgrep. I tried Yake! on Python on a file where Yake! on Rust gave the same error as mentioned above, but I don't remember the file size, it seems to be around 10 megabytes. I don't remember exactly. Personally, for me, the size of an acceptable text file is the most understandable.
A little off topic, just out of personal interest. In Yake!, there is a comprehensive assessment of keyword candidates. I would be interested in the ability to “disable” some parameters, for example, the placement of a word in the text or the capitalization of letters.

[bchigarev]

Yake! has parameters that we would like to “disable” in some experiments, such as the evaluation of a word's position relative to the beginning of the text. If the corpus consists of a set of annotations to scientific articles, this parameter will only distort the evaluation. The same may be true for words beginning with a capital letter. “Analysis of the application of secondary methods of oil recovery in this type of formation.” Here, it is better not to increase the value of the word “Analysis.” The application of Yake! gives good results, so we would like to be able to compare its main algorithm for evaluating the significance of keywords with RAKE and TextRank, even if the evaluation is subjective.
For all the methods listed, the frequency and co-occurrence of terms are important, but the assessment is constructed differently.