SentencePieceBPE + Unicode NFD preprocessing leads to noise ?

[PonteIneptique]

Hi,
I have had the issue multiple times, so I assume I am doing something wrong.

Versions:

• tokenizers==0.21.4
• transformers==4.55.4

Training script

from transformers import PreTrainedTokenizerFast
from pathlib import Path
from read import get_texts_iter_for_tokenizer
from tokenizers import SentencePieceBPETokenizer, normalizers, pre_tokenizers

def main():
    output_dir = Path("hf_tokenizer")
    output_dir.mkdir(parents=True, exist_ok=True)

    # Dump texts to a file
    texts = get_texts_iter_for_tokenizer()

    # Train SentencePiece model
    tokenizer = SentencePieceBPETokenizer()

    # Adding normalization and pre_tokenizer
    tokenizer.normalizer = normalizers.Sequence([normalizers.NFD()])
    tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()

    # Adding special tokens and creating trainer instance
    special_tokens = ["<unk>", "<pad>", "<cls>", "<sep>", "<mask>"]

    # Training from iterator REMEMBER it's training on test set...
    tokenizer.train_from_iterator(texts, special_tokens=special_tokens, show_progress=True)

    fast_tokenizer = PreTrainedTokenizerFast(
        tokenizer_object=tokenizer,
        unk_token="<unk>",
        pad_token="<pad>",
        cls_token="<cls>",
        sep_token="<sep>",
        mask_token="<mask>"
    )
    fast_tokenizer.save_pretrained(str(output_dir))

Script to reproduce bug:

from transformers import PreTrainedTokenizerFast

hf_tokenizer = PreTrainedTokenizerFast.from_pretrained("hf_tokenizer")

# Test
print(hf_tokenizer.tokenize("⁊ĩ rẽ dñi u̾sum"))
# ['âģĬ', 'i', 'Ìĥ', 'Ġre', 'Ìĥ', 'Ġdn', 'Ìĥ', 'i', 'Ġu', '̾', 'sum']
print(hf_tokenizer.decode(hf_tokenizer.encode("⁊ĩ rẽ dñi u̾sum"))
# âģĬiÌĥĠreÌĥĠdnÌĥiĠu̾sum

I assume I am doing something wrong around preprocessing / postprocessing ?

[PonteIneptique]

I replicated the thing with French training data

>>> print(fast_tokenizer.tokenize("Macquart a appelé cet insecte dacus oleæ"))
['M', 'ac', 'qu', 'art', 'Ġa', 'Ġap', 'pe', 'le', 'Ìģ', 'Ġcet', 'Ġinsecte', 'Ġdacus', 'Ġoleæ']
 >>> print(fast_tokenizer.decode(fast_tokenizer.encode("Macquart a appelé cet insecte dacus oleæ")))
'MacquartĠaĠappeleÌģĠcetĠinsecteĠdacusĠoleæ'

We can see that any decomposed diactritic is "badly" reproduced. And I don't get why.

[mikolajkniejski]

I think I know where the issue comes from. Try setting decoder for your tokenizer:

from tokenizers.decoders import ByteLevel
tokenizer.decoder = ByteLevel()

the issue is that when you use ByteLevel pretokenization respective detokenizer is not set. It is responsible for removing letters Ġ, eÃ and eÌ which are used by ByteLevel pretokenization. You have to set it manually. The default detokenizer seems to be just joining all the decoded tokens.

[mikolajkniejski]

Using the code from c0d3697 i reproduced the bug (I think)

  let mut tokenizer: Tokenizer = TokenizerBuilder::<
      BPE,
      NormalizerWrapper,
      PreTokenizerWrapper,
      PostProcessorWrapper,
      DecoderWrapper,
  >::default()
  .with_model(
      BPE::builder()
          .unk_token("[UNK]".to_string())
          .build()
          .unwrap(),
  )
  .with_pre_tokenizer(Some(PreTokenizerWrapper::ByteLevel(ByteLevel::default())))
  .build()
  .unwrap()
  .into();

  let mut trainer = tokenizer.get_model().get_trainer();
  tokenizer
      .train_from_files(&mut trainer, vec!["./data/small.txt".to_string()])
      .unwrap();

  let text = " ⁊ĩ rẽ dñi u̾sum";
  let encoding = tokenizer.encode(text, false);
  let decoding = tokenizer.decode(encoding.unwrap().get_ids(), true);
  assert_eq!(decoding.unwrap(), text);

thread 'byte_level_decoding' panicked at tests/training.rs:91:5:
assertion `left == right` failed
  left: "ĠâģĬ i Ìĥ Ġre Ìĥ Ġdn Ìĥ i Ġu ̾ sum"
 right: " ⁊i\u{303} re\u{303} dn\u{303}i u\u{33e}sum"

this can be fixed by explicitly setting decoder. It would be convenient if the respective detokenizer got set e.g. on running decode or on build but the build function uses generic types and not enums. It can also be done on type conversion from TokenizerImpl to Tokenizer or on running decode or just documented clearly. I can work on this change, I just need to know which idea seems best.