Some test datasets are unavailable

[kkkkkkon]

Thanks for your excellent work. When I was evaluating the self_reward_7b model, I found that some datasets seemed to be invisible on huggingface, such as " zli12321/mm-vet", "zli12321/pope" . How can I obtain the corresponding data set to reproduce the results?

[zli12321]

They should be available now
Pope
MM-Vet

[kkkkkkon]

Thank you very much. By the way, have you considered to release the Vision-SR1-3B model?

They should be available now Pope MM-Vet

[zli12321]

It's here. 3B is less popular though.

[kkkkkkon]

We evaluated the downloaded results and found they didn't match the results in the paper. After checking, we found that the Vision-SR1 test results and Vision-SR1-ablation were misplaced. For example, the MMMU in Vision-SR1-ablation corresponded to the results in the paper, while the MMMU in Vision-SR1 corresponded to the ablation results.

[zli12321]

thank you for pointing the issue out. Indeed there are some misplaced paths. Rerunning the MCQ now to make sure they are put under the right path.

If it is urgent, pls email us.

[zli12321]

We just confirmed MMMU acc is 55.3. Thanks for pointing out. Will update all the results accordingly.