I want to provide the Keras_hub implementation of RoFormer.

[pass-lin]

https://github.com/ZhuiyiTechnology/roformer
Roformer is a BERT-like model. It adds the now very commonly used Rope position encoding on top of BERT. In fact, this is the first practical application of Rope position encoding.
I found that Keras_hub lacks a powerful Chinese BERT-like model. And RoFormer happens to be a native Chinese BERT model, and its architecture is very similar to that of Modern BERT. This will also be helpful for future implementations related to Modern BERT.

[pass-lin]

Currently, there are two models, Bert and XLMroberta, which have Chinese and multilingual versions. However, one problem is that they have a limit on the length, making it difficult to meet the needs of many modern long-text applications.

[pass-lin]

https://github.com/ZhuiyiTechnology/roformer-v2 (Sorry, the webpage only has a Chinese interface.)
I found that RoFormer also has a more powerful v2 version, which doesn't have a corresponding implementation in HF, but its performance is better.

I tend to think we can directly implement this version, which offers a large base and a small version. Compared to previous versions, it may be more applicable.