Stopwords for mixed languages

[xamgore]

What if a text contains words in two languages? INESCTEC/yake#61

[bunny-therapist]

This is a very good question. Of course you could naively just add multiple stopwords lists together, but that risks considering some words stopwords when they are not, because they are stopwords in some other language. You could break down the text in multiple bits and process them separately, but then you need to identify the languages.

I think the complexity kind of escalates beyond what is reasonable at some point, but I am interested in any ideas.