Request for Guidance on Using a Custom Dataset with the WAI Benchmarking Framework

[SaMMyCHoo]

Hi! I’m trying to run my own dataset for testing, but I’ve encountered many difficulties when converting it into the WAI format. For example, my dataset only contains input images, corresponding intrinsic and extrinsic parameters, and a point cloud file, which makes the benchmarking process quite challenging. Although I’ve managed to get the code running, the results are clearly incorrect, and I’m not sure where the problem lies. Therefore, I was wondering if there’s any guidance or documentation available on how to properly convert and use a custom dataset. Thank you very much!

[SaMMyCHoo]

BTW, during the process, I ran my task in images_only mode. However, I noticed that modifying the structure of the depth_map unexpectedly changes the resulting point cloud, which seems inconsistent with the definition of images_only. Do you have any idea what might be causing this?

[zhoujiawei3]

+1

[Nik-V9]

Hi, thanks for trying out the WAI format!

Please note that our current WAI based benchmarking expects the scenes to have intrinsics, extrinsics and depth. Not sure how you are dealing with this issue.

For converting a custom dataset to WAI format, I would recommend looking at the existing conversion scripts. The key thing is that WAI format expects camera poses to be in OpenCV cam2world format. Apart from this, everything should be straightforward. You can also see this PR as a nice example: #36

Not sure why the depth is influencing the images_only mode. This should not be happening since the model will only use the images.

[SaMMyCHoo]

Thanks for your replying! Now we already find the problems with the depth: it does not influence the results. The intrinsics and extrinsics is correctly processed, but our dataset does not have a depth infomation, only a point cloud data. We want to know that if we want to finetune the map-anything model on our dataset, how can we generate the depth data through the pointcloud information?