Suggestions for point cloud matching

[lortane]

Hi,

I am new to artificial vision and i am trying to segment this point cloud so i can match the parts on top of the table to some 3d models.

I have tried the following, but the results don't seem too good in my opinion.

# Downsample the point cloud
voxel_size = 10
camera_pcd = camera_pcd.voxel_down_sample(voxel_size=voxel_size)
o3d.visualization.draw_geometries([camera_pcd], window_name="Downsampled point cloud", width=800, height=600)

# Estimate normals
radius_normal = voxel_size * 2
camera_pcd = camera_pcd.estimate_normals(search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=radius_normal, max_nn=30))
camera_location = [0, 0, 0]
camera_pcd = camera_pcd.orient_normals_towards_camera_location(camera_location)
o3d.visualization.draw_geometries([camera_pcd], window_name="Estimated normals", width=800, height=600, point_show_normal=True)

# Detect planes
assert(camera_pcd.has_normals())

oboxes = camera_pcd.detect_planar_patches(
    normal_variance_threshold_deg=60,
    coplanarity_deg=75,
    outlier_ratio=0.75,
    min_plane_edge_length=0,
    min_num_points=0,
    search_param=o3d.geometry.KDTreeSearchParamKNN(knn=30))

# outlier_cloud = remove_oboxes(camera_pcd, oboxes)

geometries = []
geometries.append(camera_pcd)

print(f"Detected {len(oboxes)} planes")

for obox in oboxes:
    obox.color = [1, 0, 0]
    geometries.append(obox)

# Remove the detected planes
inlier_indices = []
for obox in oboxes:
    indices = obox.get_point_indices_within_bounding_box(camera_pcd.points)
    inlier_indices.extend(indices)

inlier_indices = list(set(inlier_indices))  # Remove duplicates
outlier_cloud = camera_pcd.select_by_index(inlier_indices, invert=True)
    

this is the result. as you can see, the plane removal eliminates points from the part on top of the table, and leaves points from the table behind.

any suggestion would be highly appreciated. as i said im new to this, so I not even sure if my approach makes sense

[RubensBenevides]

Instead of using the detect_planar_patches function, try using RANSAC's plane detection, it's simpler and fulfills your purpose:
plane_model, inliers = pcd.segment_plane(distance_threshold=0.01, ransac_n=3, num_iterations=1000).
(https://www.open3d.org/docs/release/python_api/open3d.geometry.PointCloud.html#open3d.geometry.PointCloud.segment_plane)

Also, is that voxel in centimeters? Because 10 is too big if it's in meters, try using 0.01 m (1 cm) if your cloud radius is < 3 meters.
When it comes to comparing point clouds, you'll have to register one in the other (soure->target) using a registration algorithm and evaluate one of the two metrics that open3D provides as a result, which are: RMSE of the inliers (the lower, the better) and Fitness (the higher, the better). Both metrics depend on the size of the sampling voxel and the maximum distance used to establish correspondences. Higher voxel sizes used in the down sampling stage will return high RMSE. The RMSE is the square root of the sum of the squares of the distances between the points of the two clouds within the max_correspondence_distance area. Fitness is the ratio inlier_correspondences / points_in_source, it measures the overlap. If you increase the max_correspondence_distance, the Fitness will increase, but also the RMSE. If you decrease the max_correspondence_distance, the RMSE tends to decrease, but also the Fitness. The Fitness value is bounded in [0,1] interval, but will only reaches 1.0 if the point clouds have 100% overlap. So the maximum fitness value depends on the geometry of the registered pair.

An article of mine on StackOverflow can help you with this: https://stackoverflow.com/questions/66866952/open3d-compute-distance-between-mesh-and-point-cloud/67030606#67030606

When comparing generic point clouds, it is very difficult to tell whether one has been correct aligned with the other (sucessfull registration), unless we are talking about trivial geometries. For example, it's possible that the registration result returns a transformation that whem applied to the source cloud will return a low RMSE (close or bellow the voxel size used to downsample) and a high Fitness, but the result of the registration is totally wrong. This happens a lot with indoor clouds, because the symmetry of a room allows two clouds to fit together very well, even if one is rotated 180 degrees in relation to the other.

Translated with DeepL.com (free version).

[lortane]

Hi Rubens,

Thank you so much for the answer. The voxel size is in mm, my image is about 1500mm radious.

I have used a different approach to remove the background that i have getting better results. Since I have the cad model of the bin where the parts are located and it's approximate location, I have used that to remove it by applying a rough transformation and then ICP for refining the alignment. I have been getting pretty good results so far.

The problem I am facing now is, once I have the parts isolated from the background, I am having issues identifying the parts in my point cloud. For example, if I have part A and B in the point cloud and I want to find A, if I apply FPR or RANSAC for global registration, most of the times gives a result that is completely wrong, since it considering the point for both part A and B in the point cloud. I have tried using clustering, but the results are unreliable and very dependant on the parameter tunning. Do you have any suggestions for this?

Thanks in advance! Really good post in stack overflow btw, I'll be using some of your visualization ideas in my project.

[RubensBenevides]

I didn't quite understand your problem, but I think I can help with the fine tuning of the FGR, as I've tested all kinds of configurations of this algorithm. I'm currently writing an article on the plausibility of using it to register point clouds from Terrestrial/Topographic Laser Scanners. Well, since you said that most of the time the result works, at least once it worked. If that time was using FGR, then all we need to do is fine-tune it so that it works every time, because the algorithm is very stable.

[RubensBenevides]

Please tell me which parameters you are using. Specifically, we have six parameters:

1 - voxel_size (pre-processing parameter);
2 - normal_neighborhood (pre-processing parameter);
3 - FPFH_neighborhood (pre-processing parameter);
4 - Max_Correspondence_Distance (FGR parameter match control);
5 - Tuple_Scale_Ratio (FGR parameter for match control);
6 - Division_Factor (FGR parameter for graduated non-convexity);

Additionally, you can filter the cloud with the SOR filter, which requires two more parameters, but I think your clouds are relatively clean, so you don't do that.