Paper: (https://arxiv.org/pdf/2310.07602.pdf)

News

[2023.1.15] We are very sorry for the damage to some files in the published data, and the partitioning issue of imagesets has been corrected. And you can replace these images(006702.png, 006708.png, 006710.png, 006713.png, 006717.png) in the original dataset.

[2023.10.29] We have released the dataset download link.

[2023.12.10] Our Code currently supports VFF and M2Fusion.

[2023.10.27] Our Code currently supports some baselines including Voxel RCNN, Second, Pointpillars, RDIOU. Other baselines will be updated soon.

[2023.10.15] Our code and data are still being maintained and will be released soon.

1. Introduction

Dual-Radar is a new dataset based on 4D radar that can be used for studies on 3D object detection and tracking in the field of autonomous driving. The perception system of ego vehicle includes a high-resolution camera, a 80-line LiDAR and two up-to-date and different models of 4D radars operating in different modes(Arbe and ARS548). The dataset comprises of raw data collected from ego vehicle, including scenarios such as urban and tunnels, with weather conditions of rainy, cloudy, sunny and so on. Our dataset also includes data from different time periods, including dusk, nighttime, and daytime. Our collected raw data amounts to a total of 12.5 hours, encompassing a total distance of over 600 kilometers. Our dataset covers a route distance of approximately 50 kilometers. It consists of 151 continuous time sequences, with the majority being 20-second sequences, resulting in a total of 10,007 carefully time-synchronized frames.

a) Ego vehicle's work scenario

b) Data projection visualization

Figure 1. Data collection vehicle and data projection visualization

Sensor Configuration

Our ego vehicle’s configuration and the coordinate relationships between multiple sensors are shown in Figure. 2. The platform of our ego vehicle system consists of a high-resolution camera, an new 80-line LiDAR, and two types of 4D radar. All sensors have been carefully calibrated. The camera and LiDAR are mounted directly above the ego vehicle, while the 4D radars are installed in front of it. Due to the range of horizontal view limitations of the camera and 4D radars, we only collect data from the front of our ego vehicle for annotation. The ARS548 RDI captures data within approximately 120° horizontal field of view and 28° vertical field of view in front of the ego vehicle, while the Arbe Phoenix, operating in middle-range mode, collects data within a 100° horizontal field of view and 30° vertical field of view. The LiDAR collects around the ego vehicle in a 360° manner but only retains the data in the approximate 120° field of view in front of it for annotation.

Figure 2. Sensor Configuration and Coordinate Systems

• The specification of the autonomous vehicle system platform. Our proposed dataset is collected from a high-resolution camera, an 80-line mechanical LiDAR, and two types of 4D radars, the Arbe Phoenix and the ARS548 RDI radar. Our dataset provides GPS information for timing implementation. The sensor configurations are shown in Table 1.

Table 1. The specification of the autonomous vehicle system platform

Sensors	Type	Resolution			Fov			FPS
		Range	Azimuth	Elevation	Range	Azimuth	Elevation
camera	acA1920-40uc	-	1920X	1200X	-	-	-	10
LiDAR	RS-Ruby Lite	0.05m	0.2°	0.2°	230m	360°	40°	10
4D radar	ARS 548RDI	0.22m	1.2°@0...±15° 1.68°@±45°	2.3°	300m	±60°	±4°@300m ±14°@<140m	20
	Arbe Phoenix	0.3m	1.25°	2°	153.6m	100°	30°	20

• The statistics of number of points cloud per frame. In addition,we analyze the distribution density of the point clouds and the number of point clouds per frame, as shown in Table 2.

Table 2. The statistics of number of points cloud per frame

Transducers	Minimum Values	Average Values	Maximum Values
LiDAR	74386	116096	133538
Arbe Phnoeix	898	11172	93721
ARS548 RDI	243	523	800

2. Data Acquisition Scenario

• The visualization of raw data collected under different weather conditions and the visualization of annotated 3D bounding boxes are performed separately. The three types of data(Lidar, Arbe and Ars548) are transformed onto a unified coordinate system.

a) Sunny,Daytime	b) Sunny,Nightime
c) Rainy,Daytime	d) Cloudy,Daytime

Figure 3. Visualization of raw data sequences under different weather conditions. On the left is a color RGB image, while on the right side, the cyan represents the Arbe point cloud, the white represents the LiDAR point cloud, and the yellow represents the ARS548 point cloud.

Figure 4. Visualization of 3D bounding box projection on data. The first column represents the 3D frame markers on the image. The Column 2, 3, and 4 represent the point cloud from Lidar, Arbe Phoenix radar point cloud, and ARS548 RDI radar point cloud, respectively. Each row represents a scenario type.(a): downtown daytime normal light. (b): downtown daytime backlight. (c): downtown dusk normal light. (d): downtown dusk backlight. (e): downtown clear night. (f): downtown daytime cloudy. (g): downtown rainy day. (h): downtown cloudy dusk. (i): downtown cloudy night. (j): downtown rainy night. (k): daytime tunnel. (l): nighttime tunnel.

3. Download Link

• Our dataset is freely available to researchers. Please download and sign our agreement and send it to the provided email address ([email protected]). You will receive the download link within one week.

• When unzipping the data, please file and organize it by following the format below:

    └─Dual Radar
    ├─ImageSets.zip
    ├─testing
    │  ├─testing_arbe.zip
    │  ├─testing_ars548.zip
    │  ├─testing_calib.zip
    │  ├─testing_image.zip
    │  ├─testing_label.zip
    │  ├─testing_robosense.zip
    ├─training
    │  ├─training_arbe.zip
    │  ├─training_ars548.zip
    │  ├─training_calib.zip
    │  ├─training_image.zip
    │  ├─training_label.zip
    │  ├─training_robosense.zip
    └─README_dual_radar.txt

• This folder contains 10007 frames of labeled pointclouds and image data. The structure of the folder is shown as blow:

    └─Dual Radar
    ├─ImageSets
    │      test.txt
    │      train.txt
    │      trainval.txt
    │      val.txt
    ├─testing
    │  ├─arbe
    │  │      000000.bin	# Raw pointclouds (removed None) of the Arbe.
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin	# Raw pointclouds (removed None) of the ARS548.
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png	# Undistort images of the camera.
    │  │      ...............
    │  ├─label
    │  │      000000.txt	# Label in txt format, explain later.
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin	# Raw pointclouds (removed None) of the LiDAR.
    │  │      ...............
    ├─training
    │  ├─arbe
    │  │      000000.bin
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png
    │  │      ...............
    │  ├─label
    │  │      000000.txt
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin
    │  │      ...............
    └─README.txt

4. The Description of Calib Format

• The calib.txt contains tree parts. The dataset consists of two parts: the data part and the alignment calibration file. The data part is image data in png format and point cloud data in bin format. The alignment calibration file includes calibration parameters for the four sensors. The camera-LiDAR, camera-4D radar joint calibration are shown here as examples for illustration.

   Dual Radar_cam.Intrinsics.RadialDistortion: Barrel distortion of Dual Radar_cam [ k1, k2, k3 ]
   Dual Radar_cam.Intrinsics.TangentialDistortion: radial distortion of Dual Radar_cam [ p1, p2 ]
   Dual Radar_cam.IntrinsicMatrix: Dual Radar_cam's intrinsic matrix [ af, 0, 0; 0, bf, 0; u, v, 1]
   Dual Radar_LiDAR-->Dual Radar_cam: Dual Radar_lidar to Dual Radar cam's single response matrix P(4×4)
   Dual Radar_radar--> Dual Radar_cam: Dual Radar_radar to Dual Radar_cam rotation matrix + translation matrix P(3×4)

5. Label Files Discription

• All values (numerical or strings) are separated via spaces, each row corresponds to one object. The 19 columns represent:

  Value       Name             Description
  -------------------------------------------------------------------------------------------------------
  1        type               Describes the type of object: 'Car', 'Van', 'Truck', 'Pedestrian',  'Person_sitting', 'Cyclist', 'Tram', 'Misc' or 'DonCare'
  1        truncated          Float from 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries
  1        occluded           Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly ccluded, 2 = largely occluded, 3 = unknown
  1        alpha              Observation angle of object, ranging [-pi..pi]
  4        bbox               2D bounding box of object in the image (0-based index): contains left, top, right, bottom pixel coordinates. 
  3        dimensions         3D object dimensions: height, width, length (in meters).
  3        location           3D object location x,y,z in camera coordinates (in meters).
  1        rotation_y         Rotation ry around Y-axis in camera coordinates [-pi..pi].
  1        score              Only for results: Float,indicating confidence in detection, needed for p/r curves , higher is better.
  1        track_id           Path tracking of the same object

• Since the labeling work is done in label coordinate, the bounding box out of the image FOV(1920×1080) needs to be cut.

• Location mean the xyz in label coordinate. The same coordinate origen and the relation of axis is shown below.

Figure 5. Illustration of sensor coordinate systems

• The difference between rotation_y and alpha are, that rotation_y is directly given in camera coordinates, while alpha also considers the vector from the camera center to the object center, to compute the relative orientation of the object with respect to the camera. For example, a car which is facing along the X-axis of the camera coordinate system corresponds to rotation_y=0, no matter where it is located in the X/Z plane (bird's eye view), while alpha is zero only, when this object is located along the Z-axis of the camera. When moving the car away from the Z-axis, the observation angle will change.

6. Data Statistics

Figure 6. Distribution of weather conditions.

• We separately count the number of instances for each category in the Dual-Radar dataset and the distribution of different types of weather. About two-thirds of our data are collected under normal weather conditions, and about one-third are collected under rainy and cloudy conditions. We collect 577 frames in rainy weather, which is about 5.5% of the total dataset. The rainy weather data we collect can be used to test the performance of different 4D radars in adverse weather conditions.

Figure 7. Distribution of instance conditions.

• We also conduct a statistical analysis of the number of objects with each label at different distance ranges from our vehicle, as shown in Figure 7. Most objects are within 60 meters of our ego vehicle.

7. Getting Started

Environment

This is the documentation for how to use our detection frameworks with Dual-Radar dataset. We test the Dual-Radar detection frameworks on the following environment:

• Python 3.8.16 (3.10+ does not support open3d.)
• Ubuntu 18.04/20.04
• Torch 1.10.1+cu113
• CUDA 11.3
• opencv 4.2.0.32

Preparing The Dataset

• After all files are downloaded, please arrange the workspace directory with the following structure:

• Organize your code structure as follows

    Frameworks
      ├── checkpoints
      ├── data
      ├── docs
      ├── pcdet
      ├── output

• Organize the dataset according to the following file structure

    dual_radar
      ├── lidar
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── velodyne
        ├── testing
            ├── calib
            ├── image
            ├── velodyne
      ├── radar_arbe
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── arbe
        ├── testing
            ├── calib
            ├── image
            ├── arbe
      ├── radar_ars548
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── ars548
        ├── testing
            ├── calib
            ├── image
            ├── ars548

Requirements

• Clone the repository

 git clone https://github.com/adept-thu/Dual-Radar.git
 cd Dual-Radar

• Create a conda environment

conda create -n Dual-Radardet python=3.8.16
conda activate Dual-Radardet

• Install PyTorch (We recommend pytorch 1.10.1.)

• Install the dependencies

pip install -r requirements.txt

• Install Spconv（our cuda version is 113）

pip install spconv-cu113

• Build packages for Dual-Radardet

python setup.py develop

Train & Evaluation

• Generate the data infos by running the following command:

using lidar data
python -m pcdet.datasets.dual_radar.dual_radar_dataset create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset.yaml

using arbe data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_arbe create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_arbe.yaml

using ars548 data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_ars548 create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_ars548.yaml

• To train the model on single GPU, prepare the total dataset and run

python train.py --cfg_file ${CONFIG_FILE}

• To train the model on multi-GPUs, prepare the total dataset and run

sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}

• To evaluate the model on single GPU, modify the path and run

python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}

• To evaluate the model on multi-GPUs, modify the path and run

sh scripts/dist_test.sh ${NUM_GPUS} \
    --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}

Quick Demo

Here we provide a quick demo to test a pretrained model on the custom point cloud data and visualize the predicted results

• Download the pretrained model as shown in Table 4~8.
• Make sure you have installed the Open3d and mayavi visualization tools. If not, you could install it as follow:

pip install open3d
pip install mayavi

• prepare your point cloud data

points[:, 3] = 0 
np.save(`my_data.npy`, points)

• Run the demo with a pretrained model and point cloud data as follows

python demo.py --cfg_file ${CONFIG_FILE} \
    --ckpt ${CKPT} \
    --data_path ${POINT_CLOUD_DATA}

8. Experimental Results

Table 3. Multi-modal experimental results([email protected] 0.25 0.25)

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
VFF	camera+LiDAR	94.60	84.14	78.77	39.79	35.99	36.54	55.87	51.55	51.00	model
	camera+Arbe	31.83	14.43	11.30	0.01	0.01	0.01	0.20	0.07	0.08	model
	camera+ARS548	12.60	6.53	4.51	0.00	0.00	0.00	0.00	0.00	0.00	model
M2-Fusion	LiDAR+Arbe	89.71	79.70	64.32	27.79	20.41	19.58	41.85	36.20	35.14	model
	LiDAR+ARS548	89.91	78.17	62.37	34.28	29.89	29.17	42.42	40.92	39.98	model

Table 4. Multi-modal experimental results([email protected] 0.25 0.25)

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
VFF	camera+Lidar	94.60	84.28	80.55	40.32	36.59	37.28	55.87	51.55	51.00	model
	camera+Arbe	36.09	17.20	13.23	0.01	0.01	0.01	0.20	0.08	0.08	model
	camera+ARS548	16.34	9.58	6.61	0.00	0.00	0.00	0.00	0.00	0.00	model
M2-Fusion	LiDAR+Arbe	90.91	85.73	70.16	28.05	20.68	20.47	53.06	47.83	46.32	model
	LiDAR+ARS548	91.14	82.57	66.65	34.98	30.28	29.92	43.12	41.57	40.29	model

Table 5. Single-modal experimental results([email protected] 0.25 0.25)

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
Pointpillars	LiDAR	81.78	55.40	44.53	43.22	38.87	38.45	25.60	24.35	23.97	model
	Arbe	49.06	27.64	18.63	0.00	0.00	0.00	0.19	0.12	0.12	model
	ARS548	11.94	6.12	3.76	0.00	0.00	0.00	0.99	0.63	0.58	model
RDIou	LiDAR	63.43	40.80	32.92	33.71	29.35	28.96	38.26	35.62	35.02	model
	Arbe	51.49	26.74	17.83	0.00	0.00	0.00	0.51	0.37	0.35	model
	ARS548	5.96	3.77	2.29	0.00	0.00	0.00	0.21	0.15	0.15	model
VoxelRCNN	LiDAR	86.41	56.91	42.38	52.65	46.33	45.80	38.89	35.13	34.52	model
	Arbe	55.47	30.17	19.82	0.03	0.02	0.02	0.15	0.06	0.06	model
	ARS548	18.37	8.24	4.97	0.00	0.00	0.00	0.24	0.21	0.21	model
Cas-V	LiDAR	80.60	58.98	49.83	55.43	49.11	48.47	42.84	40.32	39.09	model
	Arbe	27.96	10.27	6.21	0.02	0.01	0.01	0.05	0.04	0.04	model
	ARS548	7.71	3.05	1.86	0.00	0.00	0.00	0.08	0.06	0.06	model
Cas-T	LiDAR	73.41	45.74	35.09	58.84	52.08	51.45	35.42	33.78	33.36	model
	Arbe	14.15	6.38	4.27	0.00	0.00	0.00	0.09	0.06	0.05	model
	ARS548	3.16	1.60	1.00	0.00	0.00	0.00	0.36	0.20	0.20	model

Table 6. Single-modal experimental results([email protected] 0.25 0.25)

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
Pointpillars	LiDAR	81.81	55.49	45.69	43.60	39.59	38.92	38.78	38.74	38.42	model
	Arbe	54.63	35.09	25.19	0.00	0.00	0.00	0.41	0.24	0.23	model
	ARS548	14.40	8.14	5.26	0.00	0.00	0.00	2.27	1.64	1.53	model
RDIou	LiDAR	63.44	41.25	33.74	33.97	29.62	29.22	49.33	47.48	46.85	model
	Arbe	55.27	31.48	21.80	0.01	0.01	0.01	0.84	0.66	0.65	model
	ARS548	7.13	5.00	3.21	0.00	0.00	0.00	0.61	0.46	0.44	model
VoxelRCNN	LiDAR	86.41	56.95	42.43	41.21	53.50	45.93	47.47	45.43	43.85	model
	Arbe	59.32	34.86	23.77	0.02	0.02	0.02	0.21	0.15	0.15	model
	ARS548	21.34	9.81	6.11	0.00	0.00	0.00	0.33	0.30	0.30	model
Cas-V	LiDAR	80.60	59.12	51.17	55.66	49.35	48.72	51.51	50.03	49.35	model
	Arbe	30.52	12.28	7.82	0.02	0.02	0.02	0.13	0.05	0.05	model
	ARS548	8.81	3.74	2.38	0.00	0.00	0.00	0.25	0.21	0.19	model
Cas-T	LiDAR	73.42	45.79	35.31	59.06	52.36	51.74	44.35	44.41	42.88	model
	Arbe	22.85	13.06	9.18	0.00	0.00	0.00	0.17	0.08	0.08	model
	ARS548	4.21	2.21	1.49	0.00	0.00	0.00	0.68	0.43	0.42	model

Table 7. Single-modal experimental results in the rainy scenario([email protected] 0.25 0.25)

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
Pointpillars	LiDAR	60.57	44.31	41.91	32.74	28.82	28.67	29.12	25.75	24.24	model
	Arbe	68.24	48.98	42.80	0.00	0.00	0.00	0.19	0.10	0.09	model
	ARS548	11.87	8.41	7.32	0.11	0.09	0.08	0.93	0.36	0.30	model
RDIou	LiDAR	44.93	39.32	39.09	24.28	21.63	21.43	52.64	43.92	42.04	model
	Arbe	67.81	49.59	43.24	0.00	0.00	0.00	0.38	0.30	0.28	model
	ARS548	5.87	5.48	4.68	0.00	0.00	0.00	0.09	0.01	0.01	model

Table 8. Single-modal experimental results([email protected] 0.25 0.25) in the rainy scenario

Baseline	Data	Car			Pedestrain			Cyclist			model pth
		[email protected]			[email protected]			[email protected]
		Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
Pointpillars	LiDAR	60.57	44.56	42.49	32.74	28.82	28.67	44.39	40.36	38.64	model
	Arbe	74.50	59.68	54.34	0.00	0.00	0.00	0.32	0.16	0.15	model
	ARS548	14.16	11.32	9.82	0.11	0.09	0.08	2.26	1.43	1.20	model
RDIou	LiDAR	44.93	39.39	39.86	24.28	21.63	21.43	10.80	52.44	50.28	model
	Arbe	70.09	54.17	47.64	0.00	0.00	0.00	0.63	0.45	0.45	model
	ARS548	6.36	6.51	5.46	0.00	0.00	0.00	0.13	0.08	0.08	model

9. Acknowledgement

• Thanks for the sensor support provided by Beijing Jingwei Hirain Technologies Co., Inc.

10. Citation

• If you find this work is useful for your research, please consider citing:

@article{zhang2023dual,
  title={Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autononous Driving},
  author={Zhang, Xinyu and Wang, Li and Chen, Jian and Fang, Cheng and Yang, Lei and Song, Ziying and Yang, Guangqi and Wang, Yichen and Zhang, Xiaofei and Yang,Qingshan and Li, Jun},
  journal={arXiv preprint arXiv:2310.07602},
  year={2023}
}