issues about create_data

[sunnyHelen]

Hi, thanks for sharing your great work. I encounter some issues during creating data by running create_data.py
First
create reduced point cloud for training set
[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 24, in kitti_data_prep
kitti.create_reduced_point_cloud(root_path, info_prefix)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 374, in create_reduced_point_cloud
_create_reduced_point_cloud(data_path, train_info_path, save_path)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 314, in _create_reduced_point_cloud
count=-1).reshape([-1, num_features])
ValueError: cannot reshape array of size 461536 into shape (6)

It seems to set the num_features=4 and front_camera_id=2?
in this line:

SimIPU/tools/data_converter/kitti_converter.py

Line 291 in 5b346e3

num_features=6,

I assume doing this can solve the problem but encounter another problem when
Create GT Database of KittiDataset
[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 44, in kitti_data_prep
with_bbox=True) # for moca
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
P0 = np.array(example['P0']).reshape(4, 4)
KeyError: 'P0'

Can you help me figure out how to solve these issues?

[zhyever]

You should set front_camera_id as 0 for KITTI.

SimIPU/tools/data_converter/kitti_converter.py

Line 292 in 5b346e3

front_camera_id=0):

:D Since the released codes are only supporting pre-training on KITTI, data preparation is similar to standard mmdet3d. So, you can utilize the standard mmdet3d (correct version introduced in README.md) to run create_data.py and then link the prepared data to the simipu repo.

[sunnyHelen]

Thank you for your quick reply.
when I create GT Database of KittiDataset
[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 44, in kitti_data_prep
with_bbox=True) # for moca
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
P0 = np.array(example['P0']).reshape(4, 4)
KeyError: 'P0'

SimIPU/tools/data_converter/create_gt_database.py

Line 275 in 5b346e3

P0 = np.array(example['P0']).reshape(4, 4)

It seems no P0 key. And there are some different places compared with the mmdet3d one. How should I properly creat the data?

[zhyever]

Sorry that I missed your problems since I was busy recently. There is a problem with my last answer. You should set front_camera_id=2.

Actually, I recommend that you clone the mmdet3d and utilize the official codes to generate the KITTI dataset. You can directly link the mmdet3d-generated KITTI to the SimIPU repo.

[sunnyHelen]

Got it. Thanks for your reply.

[sunnyHelen]

But I encounter a problem when I attempt to conduct Camera-lidar fusion-based 3D object detection on kitti dataset.
I follow your instruction to do that:
bash tools/dist_train.sh project_cl/configs/kitti_det3d/moca_r50_kitti.py 8 --work-dir work_dir/

But there is a problem when loading data. Does it seem related to the data label? Could please help me?

Original Traceback (most recent call last):
File "/mnt/cache/chen/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/mnt/cache/chen/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/mnt/cache/chen/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/mnt/lustre/chen/hzha/mmdetection/mmdet/datasets/dataset_wrappers.py", line 151, in getitem
return self.dataset[idx % self._ori_len]
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/custom_3d.py", line 387, in getitem
data = self.prepare_train_data(idx)
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/kitti_dataset.py", line 122, in prepare_train_data
example = self.pipeline(input_dict)
File "/mnt/lustre/chen/hzha/mmdetection/mmdet/datasets/pipelines/compose.py", line 40, in call
data = t(data)
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/pipelines/transforms_3d.py", line 185, in call
img=img)
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 388, in sample_all
avoid_coll_boxes_2d)
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 546, in sample_class_v2
sp_boxes_2d = np.stack([i['box2d_camera'] for i in sampled],
File "/mnt/lustre/chen/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 546, in
sp_boxes_2d = np.stack([i['box2d_camera'] for i in sampled],
KeyError: 'box2d_camera'

[zhyever]

Oh, this issue is caused by the key of box2d_camera in dp_sampler. In 'tools/create_data.py', you can find the calling of create_groundtruth_database, which is used to generate the sampled objects for data augment. Since we choose the moca as our baseline method, there are tons of modifications to this ground_database generation function.

Hence, if you create the Kitti dataset via the official mmdet3d codebase, I think you should run the create_groundtruth_database function (comment other lines of code in the kitti_data_prep function) in SimIPU (or Moca) to create the sampled object dataset. If you have created the sampled object dataset via our codes, but there are still these bugs, please report to me and I will have a check. I run the codes before I push this repo to github, so there should have been OK.

[sunnyHelen]

Thanks a lot. I used the official mmdet3d to create the data label before. I'll follow your instruction to run the create_groundtruth_database function.

[sunnyHelen]

Hi. I tried to run the create_groundtruth_database function. But it seems we go back to the previous problem:

[ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/create_data.py", line 247, in
out_dir=args.out_dir)
File "tools/create_data.py", line 44, in kitti_data_prep
with_bbox=True) # for moca
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
P0 = np.array(example['P0']).reshape(4, 4)
KeyError: 'P0'

[zhyever]

Let me explain why there are problems. We first conduct experiments on KITTI dataset, where the used images come from the second camera. So, when creating the KITTI, all PX should be P2 (utilize the camera parameters from the second camera). Later, we try to do experiments on Waymo, where the utilized images are in the front view, having a number of 0. Hence, we hack the codes to generate related data with P0.

However, when I push the codes that only support KITTI, I forget to change the data-related codes to the KITTI version. So, you meet problems about KeyError: 'P0'. For KITTI, just utilize P2. :D

[sunnyHelen]

Hi, thanks for your help. I successfully created the label after changing P0-->P2.
But the error still exists when:
bash tools/dist_train.sh project_cl/configs/kitti_det3d/moca_r50_kitti.py 8 --work-dir work_dir/

Original Traceback (most recent call last):
File "/mnt/cache/chenzhuo1/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/mnt/cache/chenzhuo1/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/mnt/cache/chenzhuo1/anaconda3/envs/simipu/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/mnt/lustre/chenzhuo1/hzha/mmdetection/mmdet/datasets/dataset_wrappers.py", line 151, in getitem
return self.dataset[idx % self._ori_len]
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/custom_3d.py", line 387, in getitem
data = self.prepare_train_data(idx)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/kitti_dataset.py", line 122, in prepare_train_data
example = self.pipeline(input_dict)
File "/mnt/lustre/chenzhuo1/hzha/mmdetection/mmdet/datasets/pipelines/compose.py", line 40, in call
data = t(data)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/pipelines/transforms_3d.py", line 185, in call
img=img)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 388, in sample_all
avoid_coll_boxes_2d)
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 546, in sample_class_v2
sp_boxes_2d = np.stack([i['box2d_camera'] for i in sampled],
File "/mnt/lustre/chenzhuo1/hzha/SimIPU/mmdet3d/datasets/pipelines/dbsampler.py", line 546, in
sp_boxes_2d = np.stack([i['box2d_camera'] for i in sampled],
KeyError: 'box2d_camera'

[zhyever]

I will have a check from scratch ASAP and update this repo. Btw, that's the problem only for the Moca training (our downstream task on 3D detection). While the gt_sampler does not work, you can still run the SimIPU since our pre-training method does not need any gt information.

[sunnyHelen]

Yeah, I've tried the pretraining code, which is totally ok. Thanks for your help.

[bhavyagoyal]

Hi @zhyever, I am running into the same error (KeyError: 'box2d_camera') for the downstream evaluation on Kitti dataset. Pretraining step does not have any issue. Let me know if there is an update. Thanks for the help!

[sunnyHelen]

Hi, is there any new thing about solving the problem?

[zhyever]

Sorry for the late.

Download the pkl and the zipped gt_database.

Rename the pkl file to kitti_dbinfos_train.pkl and put it under your data folder. Unzip the .zip file, rename the folder to kitti_gt_database, and put it under your data folder.

The result can be like this:

Then, run the training script again.