feat(transformers): add flava model

[wcrzlh]

What does this PR do?

This pr is extracted from Mike's PR.

Adds # (feature)
✅ Flava Model

>>> from PIL import Image
>>> import requests
>>> from mindone.transformers import AutoProcessor, FlavaModel

>>> model = FlavaModel.from_pretrained("facebook/flava-full", revision="refs/pr/6")
>>> processor = AutoProcessor.from_pretrained("facebook/flava-full", revision="refs/pr/6")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="ms", padding=True)

>>> outputs = model(**inputs)

>>> image_embeddings = outputs.image_embeddings
>>> text_embeddings = outputs.text_embeddings
>>> multimodal_embeddings = outputs.multimodal_embeddings

>>> outputs.image_embeddings.shape
(1, 197, 768)

>>> text_embeddings.shape
(1, 7, 768)

>>> multimodal_embeddings.shape
(1, 205, 768)

version	mode	model	precision	task	s/step)	weight load(s)
ms2.7.0	pynative	FlavaModel	fp32	FeatureExtraction	0.07	15

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
  documentation guidelines
• Did you build and run the code without any errors?
• Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

[gemini-code-assist]

Summary of Changes

Hello @wcrzlh, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the mindone/transformers library by introducing the full FLAVA model architecture, adapted for MindSpore. It provides a robust framework for multimodal tasks by integrating dedicated image and text processing components, along with a comprehensive model structure that supports various pre-training objectives like Masked Image Modeling (MIM), Masked Language Modeling (MLM), Image Text Matching (ITM), and Global Contrastive Learning. The changes ensure seamless integration with existing auto-loading mechanisms and include a thorough testing suite to validate the new implementation.

Highlights

• New FLAVA Model Integration: The complete FLAVA (Fusion Language and Vision Architecture) model has been added to the mindone/transformers library, enabling advanced multimodal capabilities.
• Comprehensive Model Components: This includes the core FlavaModel, FlavaForPreTraining, FlavaImageModel, FlavaTextModel, FlavaMultimodalModel, and FlavaImageCodebook, along with their respective embeddings, encoders, and prediction heads.
• Dedicated Image Processing: New image processing utilities, FlavaImageProcessor and FlavaFeatureExtractor, are introduced to handle image inputs, including masking generation for pre-training tasks.
• Unified Processing and Auto-Class Integration: A FlavaProcessor is added to combine image and text processing, and all FLAVA components are integrated into the mindone.transformers.models.auto classes for automatic configuration and modeling.
• Extensive Testing Suite: A new testing suite (test_modeling_flava.py) has been added to ensure the correctness and compatibility of the FLAVA model components.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the FLAVA model, a new multimodal model, into the library. The changes include the model implementation, associated image processors, and updates to auto-classes for registration. My review focuses on several critical bugs found in the model's output handling, potential logic errors in loss calculation, and maintainability improvements such as correcting type hints and avoiding wildcard imports. I've also noted a typo in a test directory name.

[gemini-code-assist]

There's a potential logic bug here. When pos_pairs.any() is False (i.e., there are no positive image-text pairs in the batch), pos_mask becomes ms.tensor([True]). When this mask is used to index a batch of embeddings, it will likely select only the first element of the batch. This means subsequent losses (like MMM and global contrastive) would be computed on a potentially negative sample, which is incorrect. A better approach would be to let pos_mask be pos_pairs directly. If all values are False, the indexed tensors will be empty, and the losses should gracefully handle this (e.g., result in zero loss).

[gemini-code-assist]

There seems to be a typo in the directory name for this test file. It is named fllava, but it should probably be flava to match the model name. Please correct the directory name for consistency and to ensure tests are discovered correctly.

[wcrzlh]

fixed

[zhtmike]

LGTM