About running train_nn.py and the dataset

[yuanxu-eth]

Dear Mr Daniel,
I’m XU, a student in Tsinghua Univ, China. I want to ask about your work “GelSight Simulation for Sim2Real Learning”. Your ideas are very wonderful and the experimental steps are carefully designed. But when I run the code, I meet some problem.
1.Some packages are not found when running experiments.sim2real.train_nn.py, such as the “ConfusionMatrixPlotter”, ”SimulationApproach”, “smartlab_gelsight2014_config”.
2.In your dataset, some sim figure(left one) are much shallow than the corresponding real one (right one), which may impair the performance. And I don't know what causes the non-corresponding.

I didn’t find your email so I send this message. Thank you very much if you can give me feedback.

yours

[danfergo]

Hi XU,
thank you for your feedback.

1. Are you running it as ./experiments.sim2rel.train.py? Maybe you want to run the script as python -m experiments.sim2real.train_nn (it should be python version 3) . The files seem to be in the experiments/util/ folder, I don't know what else could be.
2. Indeed, there are some differences. You can try to better adjust the simulation model to better satisfy your needs. The model contains many parameters that affect the geometry.. the number of smoothing filters applied, the px2cm ratio, etc. Our experiments assumed that there would be always some gap between the simulated images and the real images, and we bridged this gap by introducing randomly distributed noise on the simulated elastomer heightmap. Possibly you are referring to our older shorter paper, but we have recently published a more recent version of this work https://arxiv.org/pdf/2101.07169.pdf all the materials are also referred at https://danfergo.github.io/gelsight-simulation/