Code

Alex lemmatizer classifier 2#1422

Code

Merged

AngledLuffa merged 17 commits intodevstanfordnlp/stanza:devfrom alex_lemmatizer_classifier_2stanfordnlp/stanza:alex_lemmatizer_classifier_2Copy head branch name to clipboardDec 23, 2024

+2,332-6

Commits

Commits on Dec 22, 2024

• Binary (or n-way) classifier on top of the standard lemmatizer for 's token in English or other lemmas with ambiguous resolutions

  Show description for 653c324
  

  SecroLoL

  authored and

  AngledLuffa

  committed
  653c324

  Copy full SHA for 653c324

• Write data in json instead of one line per datum. Will make it easier to store other information with the data, such as the tag being processed

  AngledLuffa

  committed
  4766f7e

  Copy full SHA for 4766f7e

• Wrapper file for training the lemma classifier. Will download missing charlms if they exist

  Show description for b674892
  AngledLuffa

  committed
  b674892

  Copy full SHA for b674892

• Add a script to convert the various datasets to a lemma classifier dataset

  Show description for b5f8cf0
  AngledLuffa

  committed
  b5f8cf0

  Copy full SHA for b5f8cf0

• Add Greek as an option to the lemma_classifier data preparation

  Show description for 9917cc4
  AngledLuffa

  committed
  9917cc4

  Copy full SHA for 9917cc4

• Add a test of the lemma_classifier data preparation code

  Show description for 68d7e87
  AngledLuffa

  committed
  68d7e87

  Copy full SHA for 68d7e87

• Add a test which iterates the LSTM and transformer versions of the LemmaClassifier model

  Show description for f2317ef
  AngledLuffa

  committed
  f2317ef

  Copy full SHA for f2317ef

• Add utility to train multiple file variants at the same time

  

  SecroLoL

  authored and

  AngledLuffa

  committed
  828e02e

  Copy full SHA for 828e02e

• Add a flag to the classifier training scripts to force overwriting an existing model

  AngledLuffa

  committed
  ff0103c

  Copy full SHA for ff0103c

• Keep the target words (words which we trained to recognize) in a set. Will be useful for integrating with the Pipeline

  Show description for 89a8965
  AngledLuffa

  committed
  89a8965

  Copy full SHA for 89a8965

• Add a short script to attach a LemmaClassifier to a Lemmatizer trainer

  AngledLuffa

  committed
  3be9086

  Copy full SHA for 3be9086

• Integrate contextual classifiers in the lemmatizer into the Pipeline - now running on text with a lemma trainer that has one or more of these classifiers should attach the words correctly

  AngledLuffa

  committed
  e2feebb

  Copy full SHA for e2feebb

• Data preparation script returns the prepared sentences & doesn't necessarily require the sentences be written anywhere

  AngledLuffa

  committed
  037633f

  Copy full SHA for 037633f

• Build a lemma classifier from a variety of English datasets. When including all of them, there seems to be enough 's -> have without adding artificial data

  AngledLuffa

  committed
  0e67db7

  Copy full SHA for 0e67db7

• Make prepare_lemma_treebank automatically prepare the lemma_classifier data, make run_lemma automatically attach it

  AngledLuffa

  committed
  82c70f6

  Copy full SHA for 82c70f6

• Decide whether or not to use lemma_classifier based on charlm args, if lemma_classifier is not specifically set. Pass along the charlm args to the lemma classifier as well

  AngledLuffa

  committed
  6529df0

  Copy full SHA for 6529df0

• Refactor using the contextual lemmatizer so that the training script can evaluate that lemmatizer as well

  AngledLuffa

  committed
  20c76c0

  Copy full SHA for 20c76c0