Eval code

[nisargshah1999]

Hi,

Thanks for the interesting work! I was wondering if you could share the pipeline code used for evaluation. For fair comparison, I’d like to follow the procedure described in your paper, as mentioned 'employing Qwen2.5-32B-Instruct [51] to evaluate answer correctness by comparing extracted responses with ground-truth answers (often involving complex mathematical expressions).'

[waltonfuture]

Thank you for your interest in our project! I've uploaded the evaluation code to ModelScope: https://modelscope.cn/datasets/waltonfuture/MM-Eval. After downloading and unzipping the dataset, simply run bash run.sh to get started. Feel free to use it! There might be minor bugs related to path configurations, so please pay attention to file path settings when using it.

[ppalantir]

@waltonfuture
I tested the released model in https://huggingface.co/WaltonFuture/Qwen2.5-VL-7B-MM-UPT-MMR1, but the results are lower than those reported in the paper — 70.2 vs 72.9 on MathVista, and 67.24 vs 68.74 on We-Math.

Could you please explain what might account for this gap?

[waltonfuture]

I think that this may be due to different machines (we use h100 for testing).

[ppalantir]

@waltonfuture Are there any plans to release more geometry or GeoQA-trained models in the future? Thanks!