Unable to reproduce ETH3D results

[drkoller]

Thank you for making your code available!

I am trying to use your code to reproduce the reconstruction results that are posted on the ETH3D website for ACMMP as of 30-April-2021.

When I ran the code in this repository on the ETH3D high-res multi-view training dataset images to make 3D point clouds and then computed the F₁ scores at 2 cm tolerance, I got following results:

Dataset	F1 score (GitHub code)	F1 score (ETH3D website)	Δ
courtyard	85.52	88.47	-2.95
delivery	83.42	88.49	-5.07
electro	85.99	88.31	-2.32
facade	69.20	72.09	-2.89
kicker	79.36	83.42	-4.06
meadow	66.14	72.77	-6.63
office	76.94	82.65	-5.71
pipes	80.15	78.22	+1.93
playground	71.62	75.18	-3.56
relief	82.66	86.24	-3.58
relief_2	81.82	85.76	-3.94
terrace	88.90	90.74	-1.84
terrains	89.03	92.13	-3.10
Mean	80.06	83.42	-3.36

I realize that the algorithm is non-deterministic, and that F₁ scores may vary slightly from one execution to the next. However, the F₁ scores for almost all of the datasets are significantly worse when I run this ACMMP code, compared to the results posted on the ETH3D website.

What are the differences between the code posted in this repository and the code that was used to generate the ACMMP results posted on the ETH3D website?

[GhiXu]

Thanks for your feedback. At first, I wonder if the code cleaning affected the results. Thus, I restest the code on ETH3D high-res training dataset and get the following results:

Mean	courtyard	delivery_area	electro	facade	kicker	meadow	office	pipes	playgroud	relief	relief_2	terrace	terrains
83.67	89.26	87.79	88.38	73.24	83.26	71.46	84.23	81.94	75.99	85.67	85.09	90.32	91.07

Although the algorithm is non-deterministic, the above results are almost the same as those posted on the ETH3D website. I think maybe the difference in our input leads to our different results.

[GhiXu]

@drkoller I have checked the pair.txt files you provided in ACMP, your pair.txt files are the same as mine. Can you provide some cam.txt files for me?

[drkoller]

@GhiXu, thanks for your willingness to look into this. I am attaching here a subset of my input files for the kicker and meadow datasets, including several cam.txt files:

kicker-pair.txt
kicker-00000000_cam.txt
kicker-00000001_cam.txt
kicker-00000002_cam.txt
kicker-00000003_cam.txt
kicker-00000000.jpg
meadow-pair.txt
meadow-00000000_cam.txt
meadow-00000001_cam.txt
meadow-00000002_cam.txt
meadow-00000003_cam.txt
meadow-00000000.jpg

I will be anxious to hear if you see any differences betwen my input files and your own!

[fouralarmclock]

Hi~ Thanks for sharing your excellent work on ACMMP. I met the same problem with @drkoller . I am sure the reason is the sfm results. Could u provide your sfm results for us? Thank u very much