Skip to content

Conversation

zhtmike
Copy link
Collaborator

@zhtmike zhtmike commented Sep 17, 2025

What does this PR do?

Fixes # (issue)

Adds # (feature)
Add Lxmert model

>>> from transformers import AutoTokenizer
>>> from mindone.transformers import LxmertForQuestionAnswering
>>> import mindspore as ms

>>> tokenizer = AutoTokenizer.from_pretrained("unc-nlp/lxmert-base-uncased", revision="refs/pr/3")
>>> model = LxmertForQuestionAnswering.from_pretrained("unc-nlp/lxmert-base-uncased", revision="refs/pr/3")

>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

>>> inputs = tokenizer(question, text, return_tensors="np")
>>> for k, v in inputs.items():
...     inputs[k] = ms.tensor(v)

>>> outputs = model(**inputs)

>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()

>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)

>>> # target is "nice puppet"
>>> target_start_index = ms.tensor([14])
>>> target_end_index = ms.tensor([15])

>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.loss
>>> round(loss.item(), 2)

will encounter same error as huggingface/transformers#7266 (comment)

need to migrate demo to validate end-to-end accuracy: https://github.com/huggingface/transformers-research-projects/tree/main/lxmert

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline?
  • Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
    documentation guidelines
  • Did you build and run the code without any errors?
  • Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

@zhtmike zhtmike added the feature request Add new features label Sep 17, 2025
@zhtmike zhtmike marked this pull request as ready for review September 18, 2025 09:04
@zhtmike zhtmike requested a review from vigo999 as a code owner September 18, 2025 09:04
@zhtmike zhtmike added new model add new model to mindone and removed feature request Add new features labels Sep 23, 2025
@zhtmike zhtmike self-assigned this Sep 23, 2025
@zhtmike
Copy link
Collaborator Author

zhtmike commented Sep 23, 2025

/gemini review

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the Lxmert model, a multi-modal transformer for language and vision, ported from Hugging Face's PyTorch implementation to MindSpore. The core model logic and auto-model registration are included. However, the review identified several critical issues stemming from the direct porting of PyTorch-specific code, which will lead to runtime errors. These include incorrect weight initialization and tensor manipulation syntax. Additionally, the accompanying tests are incomplete, covering only the base model and not the heads for pre-training or question answering. I have provided detailed comments and suggestions to address these issues, fix the bugs, and enhance the overall code quality and test coverage.

@vigo999 vigo999 added this to mindone Sep 29, 2025
@vigo999 vigo999 moved this to In Progress in mindone Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model add new model to mindone

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant