feats(transformers):add bitnet model

[iugoood]

Add

1 add bitnet model
2 add UT

ps: Quantitative weights cannot be validated.

Usage

from transformers import AutoTokenizer
from mindone.transformers import BitNetForCausalLM
import mindspore as ms
# Quantized weights are temporarily unsupported.
model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T")
tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T")
prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: '
inputs = tokenizer(prompt, return_tensors="np")
# Generate
generate_ids = model.generate(ms.tensor(inputs.input_ids), max_length=100)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

Performance

Experiments are tested on Ascend Atlas 800T A2 machines with mindspore 2.6.0.

model	mode	speed
bitnet	pynative	2.41 token/s

[gemini-code-assist]

Summary of Changes

Hello @iugoood, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the BitNet model into the mindone/transformers framework. It provides the full implementation of the BitNet architecture, including its core components like RMSNorm, MLP, Attention, and Rotary Embedding, adapted for MindSpore. The integration also extends to the auto-configuration and auto-modeling systems, making the BitNet model readily available for use within the library. Additionally, a robust test suite has been added to validate the MindSpore implementation against its PyTorch counterpart.

Highlights

• New Model Integration: The BitNet model, including BitNetModel, BitNetPreTrainedModel, and BitNetForCausalLM, has been added to the mindone/transformers library.
• Auto-Configuration Support: The new BitNet model is now integrated into the automatic configuration and modeling utilities, allowing for easier instantiation and usage.
• Comprehensive Testing: Dedicated unit tests for the BitNet model have been introduced, ensuring functional correctness and compatibility by comparing MindSpore and PyTorch outputs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds the BitNet model. The implementation looks solid, but there are a few areas for improvement. The model currently depends on an un-vendored BitNetConfig from the transformers library, which should be included for self-containment. The tests are incomplete, lacking coverage for BitNetForCausalLM and support for MindSpore's Graph mode, which is crucial for performance. Additionally, there are some minor documentation issues and a wildcard import that should be addressed for better code quality and maintainability.

[gemini-code-assist]

The model configuration BitNetConfig is imported from the transformers library, which creates an external dependency. To ensure this model is self-contained within the mindone library, please vendor the configuration_bitnet.py file into this pull request, similar to how other models are structured in this repository.

[gemini-code-assist]

The test currently only runs in PyNative mode (mode=1). MindSpore's Graph mode (mode=0) is crucial for performance. Please ensure the model supports Graph mode and add it to the test modes, e.g., MODES = [0, 1].

[gemini-code-assist]

The test suite is missing a test case for BitNetForCausalLM. It's important to test the full model, including the language modeling head, to ensure its correctness. Please add a new test case to BITNET_CASES (after renaming GEMMA3_CASES) for BitNetForCausalLM.

[gemini-code-assist]

Wildcard imports (*) can lead to namespace pollution and make code harder to read and maintain. It's better to explicitly import the classes you need.

Suggested change

	from .modeling_bitnet import *
	from .modeling_bitnet import BitNetForCausalLM, BitNetModel, BitNetPreTrainedModel