Correct way to input camera view(s)

[jenspetersen]

Hi!

Quick question, my understanding is that ReCogDrive only uses the front view for driving, and the code says the same (seeing that the feature builder uses cam_f0). But in your paper (Figure 5) you show a front view with parts of the side views concatenated. Is this what's supposed to go into the network? Also, what aspect ratio should the vision input have? I want to use your pretrained networks, not train one myself.

Thanks!

[owl-10]

Hi!

Quick question, my understanding is that ReCogDrive only uses the front view for driving, and the code says the same (seeing that the feature builder uses cam_f0). But in your paper (Figure 5) you show a front view with parts of the side views concatenated. Is this what's supposed to go into the network? Also, what aspect ratio should the vision input have? I want to use your pretrained networks, not train one myself.

Thanks!

Hello,
Thank you for your interest in our work!

Regarding your question about the input image in ReCogDrive, on NAVSIM we use the Front View Image. The front view image is processed by InternVL, where we set max num patch = 12. Specifically, each image is divided into 448×448 patches — we have 2×4 patches plus one thumbnail of the original image, for a total of 9 patches. Each patch produces 256 tokens, which are then fed into the LLM.

In the paper, the concatenated multi-view image shown in Figure 5 is for better illustration of turning scenes. In Bench2Drive, we use multi-view images, where each image is divided into 1×2 + 1 patches before being sent to the LLM.

As for your question, the currently released model only supports the front-view input and does not support concatenated multi-view images. We may release a version trained with multi-view images on NAVSIM in the future.

InternVL itself supports arbitrary aspect ratios — the image will be preprocessed as described above: it’s first adjusted to the closest predefined ratio, then divided into patches. You can refer to the InternVL paper for more details.