Weird outputs using 'large' DiT planner

[ahmadchalhoub]

Hello,

I am doing some experiments using the ReCogDrive model with the nuScenes dataset, and I noticed some weird behavior specifically using the 'large' DiT planner on my custom implementation that integrates with nuScenes (when using self.dit_type == "large" here:

recogdrive/navsim/agents/recogdrive/recogdrive_agent.py

Line 74 in 1590c53

cfg = make_recogdrive_config(self.dit_type, action_dim=3, action_horizon=8, grpo=self.grpo, input_embedding_dim=1536,sampling_method=sampling_method)

).

I have done the integration of the ReCogDrive pipeline with the nuScenes dataset with all the needed custom work, and I am getting really good performance when I use the 'small' DiT planner (check the attached image below), with both the 2B and 8B InternVL VLMs. Below is a sample result from the 2B VLM model (with the 'small' DiT planner):

However, the 'large' DiT planner generates weird trajectories, shown below (the result below is using the 2B VLM):

I was wondering if you have any thoughts about what could be causing this? Amazing work btw!

Thanks,
Ahmad

[owl-10]

Hello,

I am doing some experiments using the ReCogDrive model with the nuScenes dataset, and I noticed some weird behavior specifically using the 'large' DiT planner on my custom implementation that integrates with nuScenes (when using self.dit_type == "large" here:

recogdrive/navsim/agents/recogdrive/recogdrive_agent.py

Line 74 in 1590c53

cfg = make_recogdrive_config(self.dit_type, action_dim=3, action_horizon=8, grpo=self.grpo, input_embedding_dim=1536,sampling_method=sampling_method)
).
I have done the integration of the ReCogDrive pipeline with the nuScenes dataset with all the needed custom work, and I am getting really good performance when I use the 'small' DiT planner (check the attached image below), with both the 2B and 8B InternVL VLMs. Below is a sample result from the 2B VLM model (with the 'small' DiT planner):

However, the 'large' DiT planner generates weird trajectories, shown below (the result below is using the 2B VLM):

I was wondering if you have any thoughts about what could be causing this? Amazing work btw!

Thanks, Ahmad

Thank you for your interest in our work. I previously encountered a similar trajectory issue, which was most likely caused by an incorrect model parameter loading. Please check the relevant parameters, such as whether dit_type is correctly set to large. You can also try setting

self.load_state_dict(filtered_ckpt, strict=True)
(see:

recogdrive/navsim/agents/recogdrive/recogdrive_agent.py

Line 100 in 1590c53

self.load_state_dict(filtered_ckpt, strict=False)

)

to verify whether all model parameters are being loaded correctly. It may also be that the models I provided consist only of VLM 2B and 8B along with a small DiT, and do not include a large DiT. I am not sure whether there was a mix-up on your side, or whether an error occurred when loading the model during evaluation after training.