Skip to content

Latest commit

 

History

History
127 lines (113 loc) · 12.4 KB

benchmark.md

File metadata and controls

127 lines (113 loc) · 12.4 KB

Benchmark

Replica

Here are the detailed comparison results on Replica.

ATE

Note: DPVO run with monocular data and requires scale correction during trajectory alignment.

Algorithm\ATE(RMSE[cm]) Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM 1.69 2.04 1.55 0.99 0.90 1.39 3.97 3.08 1.95
NICE-SLAM_X 2.38 1.78 2.38 1.82 0.71 1.97 3.45 2.29 2.09
Co-SLAM 0.70 0.95 1.35 0.59 0.55 2.03 1.56 0.72 1.00
Co-SLAM_X 0.85 1.12 1.42 0.64 0.77 1.90 1.34 0.88 1.11
Vox-Fusion 0.40 0.54 0.54 0.50 0.46 0.75 0.50 0.60 0.54
Vox-Fusion_X 0.60 0.58 0.40 0.41 0.36 0.88 0.70 0.61 0.56
Point-SLAM 0.61 0.41 0.37 0.38 0.48 0.54 0.69 0.72 0.52
Point-SLAM_X 0.50 0.44 0.37 0.32 0.50 0.55 0.65 0.49 0.47
SplaTAM 0.31 0.40 0.29 0.47 0.27 0.29 0.32 0.55 0.36
SplaTAM_X 0.39 0.27 0.27 0.47 0.30 0.39 0.48 0.63 0.40
DPVO_X 0.28 0.31 0.19 0.37 0.15 0.32 0.25 0.58 0.31

2D metrics

A single Replica dataset has a total of 2000 frames, rendering is performed every 50 frames, a total of 40 images are rendered. Calculate the 2D metrics of these rendered images and get the average results.

Algorithm Metric Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM PSNR+ 22.12 22.47 24.52 29.07 30.34 19.66 22.23 24.94 24.42
SSIM+ 0.69 0.76 0.81 0.87 0.89 0.80 0.80 0.86 0.81
LPIPS- 0.33 0.27 0.21 0.23 0.18 0.23 0.21 0.20 0.23
NICE-SLAM_X PSNR+ 22.87 25.14 24.58 29.12 30.27 23.68 23.77 26.04 25.68
SSIM+ 0.79 0.84 0.82 0.88 0.91 0.86 0.86 0.88 0.85
LPIPS- 0.44 0.36 0.33 0.32 0.27 0.30 0.26 0.30 0.32
Co-SLAM PSNR+ 27.27 28.45 29.06 34.14 34.87 28.43 28.76 30.91 30.24
SSIM+ 0.91 0.90 0.93 0.96 0.96 0.93 0.94 0.95 0.93
LPIPS- 0.32 0.29 0.26 0.20 0.19 0.25 0.22 0.23 0.25
Co-SLAM_X PSNR+ 27.23 28.80 29.18 34.11 34.94 28.48 28.90 31.09 30.34
SSIM+ 0.91 0.91 0.93 0.96 0.96 0.93 0.94 0.95 0.93
LPIPS- 0.32 0.28 0.26 0.21 0.19 0.25 0.22 0.23 0.24
Vox-Fusion PSNR+ 22.39 22.36 23.92 27.79 29.83 20.33 23.47 25.21 24.41
SSIM+ 0.68 0.75 0.80 0.86 0.88 0.79 0.80 0.85 0.80
LPIPS- 0.30 0.27 0.23 0.24 0.18 0.24 0.21 0.20 0.24
Vox-Fusion_X PSNR+ 25.44 27.05 27.45 31.83 31.70 25.76 27.12 27.28 27.95
SSIM+ 0.87 0.88 0.90 0.93 0.93 0.90 0.92 0.91 0.90
LPIPS- 0.33 0.30 0.25 0.24 0.24 0.25 0.20 0.25 0.25
Point-SLAM PSNR+ 32.40 34.08 35.50 38.26 39.16 33.99 33.48 33.49 35.17
SSIM+ 0.97 0.98 0.98 0.98 0.99 0.96 0.96 0.98 0.97
LPIPS- 0.11 0.12 0.11 0.10 0.12 0.16 0.13 0.14 0.12
Point-SLAM_X PSNR+ 32.40 32.64 34.20 37.52 38.38 32.59 32.52 32.56 34.10
SSIM+ 0.97 0.96 0.97 0.98 0.98 0.97 0.97 0.97 0.97
LPIPS- 0.10 0.11 0.11 0.08 0.10 0.13 0.11 0.13 0.10
SplaTAM PSNR+ 32.86 33.89 35.25 38.26 39.17 31.97 29.70 31.81 34.11
SSIM+ 0.98 0.97 0.98 0.98 0.98 0.97 0.95 0.95 0.97
LPIPS- 0.07 0.10 0.08 0.09 0.09 0.10 0.12 0.15 0.10
SplaTAM_X PSNR+ 33.03 33.79 35.47 38.49 39.42 32.37 30.55 32.4 34.44
SSIM+ 0.97 0.96 0.98 0.98 0.98 0.96 0.95 0.94 0.96
LPIPS- 0.07 0.10 0.07 0.08 0.10 0.10 0.11 0.16 0.09

3D metrics

Algorithm Metric Room0 Room1 Room2 Office0 Office1 Office2 Office3 Office4 Average
NICE-SLAM Depth L1[cm] - 1.81 1.44 2.04 1.39 1.76 8.33 4.99 2.01 2.97
Precision [%] + 45.86 43.76 44.38 51.40 50.80 38.37 40.85 37.35 44.10
Recall [%] + 44.10 46.12 42.78 48.66 53.08 39.98 39.04 35.77 43.69
F1[%] + 44.96 44.84 43.56 49.99 51.91 39.16 39.92 36.54 43.86
Depth L1[cm] - 2.11 1.68 2.90 1.83 2.46 8.92 5.93 2.38 3.53
Acc. [cm]- 2.73 2.58 2.65 2.26 2.50 3.82 3.50 2.77 2.85
Comp. [cm]- 2.87 2.47 3.00 2.02 2.36 3.57 3.83 3.84 3.00
Comp. Ratio[<5cm %] + 90.93 92.8 89.07 94.93 92.61 85.2 82.98 86.14 89.33
NICE-SLAM_X Depth L1[cm] - 1.93 1.58 2.64 2.31 2.15 4.54 3.45 2.39 2.62
Precision [%] + 50.28 50.62 47.47 42.7 61.1 42.7 41.96 36.2 46.62
Recall [%] + 40.53 41.14 37.96 34.2 46.8 35.0 34.51 30.1 37.53
F1[%] + 44.88 45.39 42.19 38.0 53.0 38.5 37.87 32.8 41.57
Acc. [cm]- 2.10 1.76 2.14 1.87 1.47 2.25 2.33 2.37 2.03
Comp. [cm]- 3.70 3.11 3.31 2.25 2.48 4.19 3.87 4.13 3.38
Comp. Ratio[<5cm %] + 88.47 89.99 88.39 92.7 90.61 83.3 83.9 85.17 87.81
Co-SLAM Depth L1[cm] - 1.05 0.85 2.37 1.24 1.48 1.86 1.66 1.54 1.51
Acc. [cm]- 2.11 1.68 1.99 1.57 1.31 2.84 3.06 2.23 2.10
Comp. [cm]- 2.02 1.81 1.96 1.56 1.59 2.43 2.72 2.52 2.08
Comp. Ratio[<5cm %] + 95.26 95.19 93.58 96.09 94.65 91.63 90.72 90.44 93.44
Co-SLAM_X Depth L1[cm] - 1.01 0.63 2.45 1.26 1.36 2.48 2.27 1.65 1.63
Precision [%] + 88.52 88.80 81.88 84.63 91.06 60.25 63.43 86.74 80.66
Recall [%] + 74.43 74.71 68.95 73.68 76.38 55.23 56.87 70.09 68.79
F1[%] + 80.86 81.15 74.86 78.78 83.08 57.63 59.97 77.53 74.23
Acc. [cm]- 1.61 1.30 1.55 1.33 1.03 1.75 1.97 1.76 1.53
Comp. [cm]- 3.32 2.83 2.59 1.65 2.08 3.63 3.46 3.70 2.90
Comp. Ratio[<5cm %] + 89.95 90.82 90.33 94.75 92.05 87.00 87.22 86.39 89.81
Vox-Fusion Depth L1[cm] - 1.09 1.90 2.21 2.32 3.40 4.19 2.96 1.61 2.46
Precision [%] + 75.83 35.88 63.10 48.51 43.50 54.48 69.11 55.40 55.73
Recall [%] + 64.89 33.07 56.62 44.76 38.44 47.85 60.61 46.79 49.13
F1[%] + 69.93 34.38 59.67 46.54 40.81 50.95 64.56 50.72 52.20
Acc. [cm]- 2.41 1.62 3.11 1.74 1.69 2.23 2.84 3.31 2.37
Comp. [cm]- 2.60 2.23 1.93 1.39 1.80 2.71 2.69 2.88 2.28
Comp. Ratio[<5cm %] + 92.87 93.48 94.34 97.21 93.76 90.98 90.73 89.48 92.86
Vox-Fusion_X Depth L1[cm] - 0.96 0.46 0.83 0.63 1.18 1.79 1.40 1.05 1.03
Precision [%] + 95.13 93.82 92.30 91.10 89.15 87.23 88.76 78.72 89.52
Recall [%] + 74.74 76.04 74.90 74.69 72.29 67.41 69.59 61.06 71.34
F1[%] + 83.71 84.00 82.70 82.08 79.84 76.05 78.02 68.77 79.39
Acc. [cm]- 1.51 1.22 1.30 1.16 1.01 1.47 1.68 1.81 1.39
Comp. [cm]- 3.24 2.81 2.39 1.57 2.03 3.43 3.29 3.85 2.82
Comp. Ratio[<5cm %] + 89.96 90.77 91.68 95.08 92.32 87.68 87.51 86.07 90.13
Point-SLAM Depth L1[cm] - 0.53 0.22 0.46 0.30 0.57 0.49 0.51 0.46 0.44
Precision [%] + 91.95 99.04 97.89 99.00 99.37 98.05 96.61 93.98 96.99
Recall [%] + 82.48 86.43 84.64 89.06 84.99 81.44 81.17 78.51 83.59
F1[%] + 86.90 92.31 90.78 93.77 91.62 88.98 88.22 85.55 89.77
Point-SLAM_X Depth L1[cm] - 0.27 0.22 0.55 0.26 0.47 0.53 0.49 0.27 0.38
Precision [%] + 99.63 99.54 99.24 99.53 99.62 99.14 98.83 98.91 99.30
Recall [%] + 84.84 86.05 84.37 88.65 83.51 81.18 81.05 80.62 83.78
F1[%] + 91.65 92.30 91.20 93.77 90.85 89.26 89.06 88.83 90.86
Acc. [cm]- 1.45 1.14 1.20 1.04 0.85 1.32 1.56 1.48 1.25
Comp. [cm]- 3.62 3.07 3.01 1.69 2.37 3.66 3.53 4.02 3.12
Comp. Ratio[<5cm %] + 87.60 89.13 89.43 93.00 89.42 85.85 85.56 85.23 88.15

Euroc

ATE

Note: DPVO run with monocular data and requires scale correction during trajectory alignment.

Algorithm\ATE(RMSE[cm]) MH01 MH02 MH03 MH04 MH05 V101 V102 V103 V201 V202 V203 Average
DPVO 8.7 5.5 15.8 13.7 11.4 5.0 14.0 8.6 5.7 4.9 21.1 10.5
DPVO_X 10.0 7.4 11.8 15.2 8.7 9.4 15.9 10.2 6.6 6.4 12.3 10.4