Trouble installing tokenizers (python) on AWS lambda arm64 image.

[crawftv]

It seems that a tokenizers installation pulls in 'huggingface_hub' which has a 'hf-xet' dependency which has some interesting platform specific rules which fails my build on install. a dependency resolution issue.

    "hf-xet>=1.1.3,<2.0.0; platform_machine=='x86_64' or platform_machine=='amd64' or platform_machine=='arm64' or platform_machine=='aarch64'",

I noticed that hugging face_hub is only used in 2 places in the python bindings repo.
Could this dependency moved to the testing extras, extras["testing"] ?

[ArthurZucker]

Thanks for sharing, will consider, tho this would be quite breaking for us!

[ArthurZucker]

hf-xet should have full support soon I think

[ArthurZucker]

(Yep internally asked and latest version should work!)