Drivebench inference for internvl: function "process_pil_image" unused

[CrazyElements]

Hi! I noticed that in internvl_lmdeploy.py, you defined a helper function process_pil_image to preprocess image inputs for InternVL models, but it doesn’t seem to be used anywhere in the current inference pipeline.

Could you clarify whether this preprocessing step is required for correct inference (or for matching the model’s expected input format), and if so, where it should be applied?

Also, could you share the decoding settings used in your experiments, specifically max_new_tokens, top_p, and temperature?

Thanks in advance!

[owl-10]

Hi! I noticed that in internvl_lmdeploy.py, you defined a helper function process_pil_image to preprocess image inputs for InternVL models, but it doesn’t seem to be used anywhere in the current inference pipeline.

Could you clarify whether this preprocessing step is required for correct inference (or for matching the model’s expected input format), and if so, where it should be applied?

Also, could you share the decoding settings used in your experiments, specifically max_new_tokens, top_p, and temperature?

Thanks in advance!

Hi, thank you for your interest in our work.

Regarding process_pil_image in the DriveBench inference script: you’re right — this function is currently redundant and not required for correct inference. It was originally used for image preprocessing when running InternVL with the Transformers-based inference pipeline. In the current DriveBench inference, we use lmdeploy, which relies on lmdeploy’s built-in load_image function for image handling, so this helper function is no longer needed.

As for the decoding settings: the relevant top_p and temperature configurations are defined in parse_arguments, and we use the default values specified there in our experiments.

[CrazyElements]

Hi @owl-10, thanks for your help.

I downloaded owl10/ReCogDrive-VLM-8B from HuggingFace and evaluated it on DriveBench (clean task) using a single RTX 4090 GPU. However, the performance I obtained is significantly lower than the results reported in the paper.

Environment:

• CUDA 12.0
• lmdeploy 0.12.0
• torch 2.8.0
• transformers 4.57.3
• ray 2.37.0

During inference, I observed the following warnings:

lmdeploy - WARNING - For higher performance, please install FlashAttention-3
lmdeploy - WARNING - Could not find internvl2_5 in registered models. Register internvl2_5 using the BaseChatTemplate.
lmdeploy - WARNING - Fallback to pytorch engine because turbomind engine is not installed correctly. If you insist to use turbomind engine, you may need to reinstall lmdeploy from pypi or build from source and try again. 
The tokenizer you are loading from 'recogdrive-main/models/ReCogDrive_VLM_8B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

I am wondering whether these warnings (e.g., tokenizer regex issue, missing chat template registration, PyTorch fallback, or lack of FlashAttention) could meaningfully affect model performance.

Could you please clarify:

• Whether there are specific evaluation settings, prompts, preprocessing steps, or decoding configurations required to reproduce the reported numbers?
• Whether the paper results were obtained using lmdeploy (and which backend), or a different inference setup?

Thanks in advance for your guidance.

[owl-10]

Hi @owl-10, thanks for your help.

I downloaded owl10/ReCogDrive-VLM-8B from HuggingFace and evaluated it on DriveBench (clean task) using a single RTX 4090 GPU. However, the performance I obtained is significantly lower than the results reported in the paper.

Environment:

• CUDA 12.0
• lmdeploy 0.12.0
• torch 2.8.0
• transformers 4.57.3
• ray 2.37.0

During inference, I observed the following warnings:

lmdeploy - WARNING - For higher performance, please install FlashAttention-3
lmdeploy - WARNING - Could not find internvl2_5 in registered models. Register internvl2_5 using the BaseChatTemplate.
lmdeploy - WARNING - Fallback to pytorch engine because turbomind engine is not installed correctly. If you insist to use turbomind engine, you may need to reinstall lmdeploy from pypi or build from source and try again. 
The tokenizer you are loading from 'recogdrive-main/models/ReCogDrive_VLM_8B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

I am wondering whether these warnings (e.g., tokenizer regex issue, missing chat template registration, PyTorch fallback, or lack of FlashAttention) could meaningfully affect model performance.

Could you please clarify:

• Whether there are specific evaluation settings, prompts, preprocessing steps, or decoding configurations required to reproduce the reported numbers?
• Whether the paper results were obtained using lmdeploy (and which backend), or a different inference setup?

Thanks in advance for your guidance.

Hi, thanks for your interest and careful evaluation.

The performance gap is not caused by the warnings you mentioned (FlashAttention, PyTorch fallback, tokenizer regex, or chat template). These mainly affect efficiency, not task-level performance.

The low results are due to using the wrong model weights for DriveBench.
ReCogDrive-VLM-8B was trained with a different box template:

• DriveBench expects:
  <c3,CAM_FRONT,1180.8,541.7>
• Our original training uses normalized boxes like:
  <BACK_RIGHT VIEW>[481, 489, 655, 717]

This mismatch leads to poor DriveBench performance.

We provide a DriveBench-aligned model here:
https://huggingface.co/owl10/ReCogDrive-VLM-DriveLM

For more details, please refer to:
#43 (comment)

Thanks again for the report.

[CrazyElements]

Thanks for your reply. We tried the updated model you shared. Although we still haven’t been able to reproduce the reported results, we do see an improvement compared to our previous runs.

Also, we are running on Ascend 910B, and we suspect there may be some performance variance across different hardware platforms.

One more thing we noticed (and I might be misunderstanding this part, so please correct me if I’m wrong): in lmdeploy.vl.model, the _preprocess_v1_5 method in class InternVLVisionModel seems to be written for single-image input. It sets max_num directly to max_dynamic_patch (18). However, during training with multi-image input, we use:

• max_num = max(1, self.max_dynamic_patch // num_image) (which is 3 if the input frame has 6 views).

After adjusting max_num accordingly, we observed a significant performance improvement.

[owl-10]

Thanks for your reply. We tried the updated model you shared. Although we still haven’t been able to reproduce the reported results, we do see an improvement compared to our previous runs.

Also, we are running on Ascend 910B, and we suspect there may be some performance variance across different hardware platforms.

One more thing we noticed (and I might be misunderstanding this part, so please correct me if I’m wrong): in lmdeploy.vl.model, the _preprocess_v1_5 method in class InternVLVisionModel seems to be written for single-image input. It sets max_num directly to max_dynamic_patch (18). However, during training with multi-image input, we use:

• max_num = max(1, self.max_dynamic_patch // num_image) (which is 3 if the input frame has 6 views).

After adjusting max_num accordingly, we observed a significant performance improvement.

Hello, thanks for your feedback and for trying the updated model.

I recently reran the DriveBench evaluation script and found a bug on our side. In the previous evaluation script, I mistakenly used the QA system message from the training stage. However, the correct setup should use the default DriveBench system message.

I have now fixed this issue in the evaluation script. To clarify, the results reported in our paper were obtained using the default DriveBench system message, not the training QA system message.

Sorry for the confusion caused by this mistake. Thanks again for carefully checking and sharing your observations.