misalignment in the point cloud visualization and bounding box returned from LLM

[jhxiong]

Dear author,

Thanks for sharing the excellent work.

While running the llava/eval/run_llava_3d.py on the dataset ./demo/scannet/scene0356_00 to get the bounding box of the bed and the window, I want to visualize the predictions in the scene. I used the sequences of images_tensor, depths_tensor, poses_tensor, intrinsics_tensor to get the point clouds visualization, and then visualize the bounding box from the LLM prediction.

I suspect there is a coordinate shift between the point cloud visualization and the LLM prediction, so the bounding box is not aligned with the associated objects as shown in the image below.

After viewing the code, I suspect this misalignment might come from the depth cropping and scale and how the feature is backprojected in RGBDVideoTower, while I do not find a fix for this misalignment. Can you suggest a fix for this or share a sample code to visualize the point clouds and predicted bounding box where they are alignment?

Appreciate for looking into this issue.

[ZCMax]

In the RGBDVideoTower module, the point cloud generated from posed RGB-D images will be transformed to align with the center and axis of the coordinates. You need to use the axis aligned point cloud to visualize the prediction results.

[jhxiong]

In the RGBDVideoTower module, the point cloud generated from posed RGB-D images will be transformed to align with the center and axis of the coordinates. You need to use the axis aligned point cloud to visualize the prediction results.

Can you elaborate more on the "transformed to align with the center and axis of the coordinates"? I suppose this is not related to axis_align_matrix, as I did tried with axis_align_matrix@[prediction, 1] and the result does not seem to be reasonable. Can you please share the pointer to how the transformation is performed for point cloud generated from posed RGB-D images will be transformed to align with the center and axis of the coordinates? I did not find where the transformation is performed. Really appreciate!

[ZCMax]

For each image, we generate the corresponding point cloud using the depth image and camera intrinsic, and then we use the pose information to transform the point cloud from the camera coordinate to the world coordinate. Then we use the axis_align_matrix to transform the point cloud to move the center of the point cloud of the scene to the center of the world coordinate, and rotate it to axis alignment. You can see that we use the new pose combining the original pose and the axis_align_matrix here. So, in fact, we send the axis-aligned point cloud into the model to predict the bounding box. When you are visualizing the point cloud with the predicted boxes, you need to use the axis-aligned point cloud.

Besides, you mentioned that I used the sequences of images_tensor, depths_tensor, poses_tensor, intrinsics_tensor to get the point clouds visualization, and then visualize the bounding box from the LLM prediction. I think it's a reasonable point cloud visualization approach, I'm curious about your prompt you are using.

[jhxiong]

For each image, we generate the corresponding point cloud using the depth image and camera intrinsic, and then we use the pose information to transform the point cloud from the camera coordinate to the world coordinate. Then we use the axis_align_matrix to transform the point cloud to move the center of the point cloud of the scene to the center of the world coordinate, and rotate it to axis alignment. You can see that we use the new pose combining the original pose and the axis_align_matrix here. So, in fact, we send the axis-aligned point cloud into the model to predict the bounding box. When you are visualizing the point cloud with the predicted boxes, you need to use the axis-aligned point cloud.

Besides, you mentioned that I used the sequences of images_tensor, depths_tensor, poses_tensor, intrinsics_tensor to get the point clouds visualization, and then visualize the bounding box from the LLM prediction. I think it's a reasonable point cloud visualization approach, I'm curious about your prompt you are using.

The poses_tensor used to visualize the point cloud should already include the axis_align_matrix, so the visualization should be in the axis align world coordinate. Besides, I also checked with the GT bounding box of the objects in the scanrefer data, and it exhibits a good match between the visualization and the GT bounding box (that is the LLM prediction on the scanrefer benchmark may also occur the prediction bias). That is why I suspect the prediction of the LLM essentially introduces bias of the coordinate, from which I cannot find the bias originates.

The prompt is like "What is the bounding box for the bed/window. Please output the 3D bounding box of the object and then describe the object."

Do you have a visualization code to visualize the point cloud and LLM bounding box prediction? This will be helpful to debug the visualization misalignment.

[ZCMax]

For now, if you want to generate the object bounding box and caption, you need to provide the click prompt ( 3D coordinates) to help the model output the correct 3D bounding box. I think your visualization is correct, the bounding box misalignment is due to the wrong prediction from the LLM.

[jhxiong]

Thanks for your reply. From my point of view, it may not majority originate from a wrong prediction of the LLM. Below you can see the coordinate of the book shelf in your manuscript, and the coordinate in my visualization. It seems there is a major shift in x and y of the original coordinate. (as verified with the GT annotation of the coordinate of the book shelf from the scanrefer, my visualization should have a numerical match).

I prepared more visualization to illustrate the misalignment, and I highly suspect it is originated from the inconsistency of how the model preprocess the coordinate.

When using the click coordinate from the manuscript ([-0.826, 1.552, 1.153]), the response bounding box is like:

When using the click coordinate from point cloud visualization ([-0.745, 1.170, 1.200]), the model mis-identify the object as the blue trash bin, and here is the response of the bounding box:

From here, I would suspect the coordinate in the visualization does not match with the coordinate consumed by the LLM when training the model. For reference, also attached the GT bounding box for the book shelf from scanrefer:

May I kindly ask if you have the script to reproduce the eval metric on scanrefer benchmark? With that the troubleshooting should be easier to be conducted.

[rickeyhhh]

@jhxiong Hi! I wonder if you solved this coordinate misalignment issue?

[rickeyhhh]

@ZCMax Hi! I met the same problem. The bounding box size of the model output seems to be appropriate, but the center position is incorrect, have you encountered this? How should I fix it?