First words of sentences are evaluated as PROPN and will not produce expected lemmas

[gh-andre]

I'm seeing first words of sentences being evaluated as PROPN, so they are not lemmatized as expected. Here's an example.

> d = nlp.readDoc("Lines for March are forming! Sights of London, begins on March 15th, 2025")

This produces these parts of speech. Notice that Lines and Sights are evaluated as PROPN.

> d.tokens().out(nlp.its.pos)
[
  'PROPN', 'ADP',  'PROPN',
  'AUX',   'VERB', 'PUNCT',
  'PROPN', 'ADP',  'PROPN',
  'PUNCT', 'VERB', 'ADP',
  'PROPN', 'ADJ',  'PUNCT',
  'NUM'
]

Because of this, lemmas for Lines and Sights are not derived from them being just nouns:

> d.tokens().out(nlp.its.lemma)
[
  'lines',  'for',   'march',
  'be',     'form',  '!',
  'sights', 'of',    'london',
  ',',      'begin', 'on',
  'march',  '15th',  ',',
  '2025'
]

If I lowercase the whole sentence, I lose London being evaluated as PROPN, but lemmas will be correct. These are the lemmas and parts of speech for the same sentence lowercased via .toLowerCase(). Notice that lemmas are now as expected, but London is no longer evaluated as PROPN.

> d.tokens().out(nlp.its.lemma)
[
  'line',  'for',   'march',
  'be',    'form',  '!',
  'sight', 'of',    'london',
  ',',     'begin', 'on',
  'march', '15th',  ',',
  '2025'
]
> d.tokens().out(nlp.its.pos)
[
  'NOUN',  'ADP',  'NOUN',
  'AUX',   'VERB', 'PUNCT',
  'NOUN',  'ADP',  'NOUN',
  'PUNCT', 'VERB', 'ADP',
  'NOUN',  'ADJ',  'PUNCT',
  'NUM'
]

Is there a way to parse a document in such a way that the first word of a sentence is not automatically considered a PROPN?

So far, the only solution I found is to locate the first word of each sentence and lowercase it, but this will mess up cases where the first word is a proper noun, like London is beautiful.

Any help would be appreciated. Thank you!

[rachnachakraborty]

This is a thorough analysis of the issue. WinkNLP relies on a pre-trained model plus a small set of rules. We had to introduce rules because there were limited open-source training data available under a permissive license.

One change we suggest is updating the rule that forces capitalized NOUN and ADJ tokens to be tagged as PROPN. Specifically, if you modify line feature.js:191to:

pos = posOfWLC;

you can obtain the following POS tags and lemmas:

// POS Tags
[
  'NOUN',  'ADP',  'NOUN',
  'AUX',   'VERB', 'PUNCT',
  'NOUN',  'ADP',  'PROPN',
  'PUNCT', 'VERB', 'ADP',
  'NOUN',  'ADJ',  'PUNCT',
  'NUM'
]
// Lemmas
[
  'line',  'for',   'march',
  'be',    'form',  '!',
  'sight', 'of',    'london',
  ',',     'begin', 'on',
  'march', '15th',  ',',
  '2025'
]

You may want to fork the model, apply this change, and see whether it works for your use case. Note that it may slightly reduce overall POS-tagging accuracy.

[gh-andre]

@rachnachakraborty Thank you for the detailed response. It is very much appreciated. I'll give it a try.