fix: fix entity aggregation bug for NER detection

[raphael0202]

It looks like it’s because we’re using the “FIRST” aggregation strategy, with a tokenizer that is not word-aware: we’re falling back to some heuristics (the presence of spaces before/after the word), that somehow fails here.
Indeed, XLM-RoBERTa model does not use the same tokenizer as RoBERTa, and uses an Unigram model (instead of BPE), which is not word-aware.
Another issue of the “FIRST” aggregation strategy is that the ending dot after the ingredient list is predicted as part of the ingredient list, even though it’s not in the non-aggregated prediction. By switching to “SIMPLE” strategy (a strategy without an error correction mechanism), we don’t have this issue anymore, but two subwords belonging to the same word are sometimes predicted as belonging to two entities.
A more in-depth analysis of the TokenClassificationPipeline reveals that the issue comes from the Punctuation() pre-tokenizer we added: it was not included in the original tokenizer, and the heuristic doesn’t take it into account, leading to an incorrect detection. I updated the heuristic to use the word_ids provided by the tokenizer to know whether the token is a subword or not (with respect to the pre-tokenization output).
By updating the way entities are aggregated, we don't have an issue anymore with the first aggregation strategy: we keep it as it.

[codecov]

Codecov Report

Attention: Patch coverage is 12.50000% with 14 lines in your changes missing coverage. Please review.

Project coverage is 55.77%. Comparing base (6eae9d5) to head (af4ebc1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rediction/ingredient_list/transformers_pipeline.py	12.50%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1413      +/-   ##
==========================================
+ Coverage   55.64%   55.77%   +0.12%     
==========================================
  Files          83       83              
  Lines        8232     8210      -22     
==========================================
- Hits         4581     4579       -2     
+ Misses       3651     3631      -20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.