A question about adding a custom dict.

[kmining]

hello,
I want to receive a compound word as one token, and tag the token I want with the part-of-speech.
For example, when the film title is "ปกป้องโลก", "ปกป้องโลก" is taken as one token and tagged as a "noun".

I understand that I can use pythainlp's custom_dict to solve this.

But I have two questions.

=============[question]===============
1. I want to use "attacut" among the tokenizer engines,
but if custom_dict and engine=attacut are set together, the custom dictionary is not applied.
How do I apply a custom dict while using attacut?

2. If there is a movie title called "ทะเลาะกัน", It's a verb, but I want it to be tagged as a noun.
Can you add not only words but also POS to the custom dict so that it is reflected in the result?

[wannaphong]

1. No, attacut can't apply a custom dict. The custom_dict can be provided as an argument only for newmm, longest, and deepcut engine. https://pythainlp.github.io/docs/3.0/api/tokenize.html#pythainlp.tokenize.word_tokenize
2. Now, POS can't apply a custom dict. You can create new function.

def pos_tag_new(pythai_tag,custom):
 _temp = []
 for w,t in pythai_tag:
  if w in custom.keys():
   t = custom[w]
  _temp.append((w,t))
 return _temp

custom = {"ทะเลาะกัน":"Noun"}
tag = [("พวกเรา","Noun"),("ทะเลาะกัน","Verb")]
print(pos_tag_new(tag ,custom))
# [('พวกเรา', 'Noun'), ('ทะเลาะกัน', 'Noun')]

[kmining]

Thank you for your quick reply.:)

[kmining]

@wannaphong
When I tested with the easy sentence "Eat rice(กินข้าว)", all dictionary-based tokenizers such as longest and newmm gave wrong results.
I think that "rice(ข้าว)" and "eat(กิน)" should be separated as the correct answer.

So I didn't want to use the longest, newmm engine, but I have no choice but to use the user dictionary..
Is it possible to improve it?

[bact]

You can try remove "กินข้าว" from the default dictionary.

import thai_words -> remove "กินข้าว" -> you will have a modified dict -> use this as custom_dict.

very dirty and patchy, but may help in a short term.