ADD RWKV7

[pass-lin]

RWKV7 is one of the strongest RNN models available today, and we now provide a full implementation for it in keras_hub.

📚 References

• 📄 RWKV Research Paper
• 📘 Model Docs on GitHub
• Website - https://www.rwkv.com/

---

🔗 Pre-trained Checkpoints (ModelScope)

• https://www.modelscope.cn/models/Blink_DL/rwkv-7-world
• https://www.modelscope.cn/models/RWKV/rwkv7-g1
• https://www.modelscope.cn/models/Blink_DL/rwkv-7-pile

Numerical-verification and Inference Example notebook

This is the first modern RNN architecture in keras_hub. With the resurgence of recurrent models, more pre-trained RNN backbones will follow; hence this PR also serves as a reference implementation for future work.

Current progress

• [✅] backbone implementation
• [✅] checkpoint-conversion script
• [✅] tokenizer implementation
• [✅] unit tests / examples
• [✅] complete CausalLM task wrapper
• [✅] Add document

[gemini-code-assist]

Summary of Changes

Hello @pass-lin, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the keras_hub library by integrating the RWKV7 model, a cutting-edge recurrent neural network. This addition not only provides a robust new model for users but also serves as a foundational reference implementation, encouraging the future inclusion of more modern RNN architectures within the library.

Highlights

• RWKV7 Model Integration: Introduced the RWKV7 model, a powerful RNN architecture, into keras_hub, marking a significant expansion of the library's capabilities.
• Comprehensive Implementation: The pull request includes a full implementation of the RWKV7 backbone, its dedicated tokenizer, and a causal language model wrapper.
• Foundation for Modern RNNs: This is the first modern RNN architecture added to keras_hub, serving as a reference implementation and paving the way for future recurrent model additions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This PR introduces the RWKV-7 model, a powerful RNN architecture, to keras_hub. The contribution is significant and includes the backbone, tokenizer, preprocessor, an incomplete task model, and a checkpoint conversion script. The implementation follows the modular structure of keras_hub.

However, there are several critical issues that must be addressed before this PR can be merged:

1. Missing Tests: The PR lacks unit tests for all new components. According to the contribution guidelines, testing is a mandatory requirement.[^1]
2. Incomplete CausalLM Task: The RWKV7CausalLM task model is a stub with TODOs, making it non-functional for generation.
3. Critical Bugs: There are critical bugs in the tokenizer and preprocessor implementations that will cause runtime errors.
4. Style Guide Violations: There are numerous style guide violations, including a filename typo, missing docstrings, and inconsistencies with the recommended model input structure.

I've left detailed comments on these issues. Once these are resolved, this will be a great addition to the library.

[pass-lin]

@divyashreepathihalli @mattdangerw Anybody review my code?

[pass-lin]

@sachinprasadhs

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive implementation of the RWKV-7 model, a modern RNN architecture, into keras_hub. The contribution is well-structured, following the repository's modular design with a backbone, causal LM task, preprocessor, tokenizer, and a checkpoint conversion script. The code is generally of high quality.

My review highlights a few areas for improvement. There are critical bugs in the tokenizer's asset loading and saving logic that need to be addressed. The custom layers in rwkv7_layer.py have inconsistent return signatures, which could lead to runtime errors and makes the code harder to maintain. Additionally, the test files do not use the standardized helper methods from TestCase as required by the repository's style guide, which is a significant deviation. I have also pointed out some minor typos and opportunities for code clarification.

Overall, this is an excellent and valuable addition. Addressing the feedback will improve the robustness, correctness, and maintainability of the new model.

[pass-lin]

@mattdangerw @divyashreepathihalli @sachinprasadhs
It's been almost a month, is there anyone willing to pay attention to me?

[sachinprasadhs]

Apologies for the delay in review, taking a look into this. Will add my comments.

[sachinprasadhs]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the RWKV-7 model, a modern RNN architecture, to keras_hub. The implementation is comprehensive, covering the backbone, causal LM task, preprocessor, tokenizer, and a checkpoint conversion script. The code is well-structured and follows the modular design principles of the repository.

My review focuses on ensuring adherence to the repository's style guide, particularly regarding testing practices and code style conventions. I've identified several areas for improvement:

• The testing for the new components should be updated to use the standardized helper methods from the base TestCase. Some tests also contain incorrect assertions.
• There are a few deviations from the coding style, such as the use of type hints in function signatures and a few hardcoded values that could be made more flexible.
• The backbone implementation should be updated to accept a padding_mask as input, aligning with the repository's conventions.

Addressing these points will improve the consistency, correctness, and maintainability of the new model. Overall, this is a great contribution, adding a powerful and interesting architecture to the library.

[pass-lin]

Thanks for providing the details, will again circle back with the team and get back to you on this one.

Finally, please allow me to add one more point.

From February 26, 2025, to today, RWKV-LM has gained 1,000 stars (from 13.1k to 14.1k). It has also increased by 100 stars from October 16 to today. You can see this trend of star growth at the following link.

I believe this demonstrates that RWKV is a very popular and highly active community.

[tempdragon]

BTW, RWKV is also mentioned here by the Linux Foundation.
https://lfaidata.foundation/projects/rwkv/

[divyashreepathihalli]

Thank you @pass-lin!!

[pass-lin]

Thank you @pass-lin!!

I want to know if the Keras team thinks RWKV is suitable to be merged into Keras Hub.

[sachinprasadhs]

@pass-lin , We can add it since there is already a lot of efforts involved.
I will add the review for other files.
Going forward please create an issue with the details of the model along with the download trend etc.

[sachinprasadhs]

Still there are many unresolved comments, please go though them carefully and let us know once this is ready for review again.
Also, match the coding style to Keras Hub standard implementation.
refer our Model and contribution guidelines.

[pass-lin]

Following your review, I readjusted rwkv7_tokenizer. Please note that I retained the recursion, as rashly modifying the recursive code written by the original rwkv author would be too bug-prone. Given that the trie tree's maximum depth in the rwkv vocabulary is 80, stack overflow is unlikely, so I believe it should be kept.
@sachinprasadhs

[sachinprasadhs]

Okay, also once you address any comment mark the comment as resolved

[pass-lin]

Still there are many unresolved comments, please go though them carefully and let us know once this is ready for review again. Also, match the coding style to Keras Hub standard implementation. refer our Model and contribution guidelines.

I think all the current issues have been resolved, and we can proceed to the next step.

[sachinprasadhs]

There is an option/button as "Resolve conversation" for each review, could you please click that if the comment is resolved there are 100 plus comments and many are still showing open.

[pass-lin]

There is an option/button as "Resolve conversation" for each review, could you please click that if the comment is resolved there are 100 plus comments and many are still showing open.

Ok, i have resolved all conversations

[pass-lin]

@sachinprasadhs Can you review the current code?