IMU camera extrinsic is there?

[hplegend]

"
device = device_list.get_device_by_index(i)
pipeline = Pipeline(device)

    ret : CameraParamList = device.get_calibration_camera_param_list()
    print(ret.get_camera_param(0))

"
I use above code print calibration camera, and the result as following:
<OBCameraParam depth_intrinsic < fx=504.507fy = 504.528 cx =326.349 cy=343.595 width=640 height=576 >
depth_distortion < k1=16.1704 k2=8.44953 k3=0.324469 k4=16.4805 k5=13.9441 k6=1.99604 p1=5.92792e-05 p2=-1.46558e-05 >
rgb_intrinsic < fx=498.295fy = 497.981 cx =315.49 cy=248.908 width=640 height=480 >
rgb_distortion < k1=0.076085 k2=-0.107371 k3=0.0443615 k4=0 k5=0 k6=0 p1=-0.000475228 p2=0.000330378 >
transform < rot=[0.993813, 0.000713588, -0.00148801, -0.00054392, 0.993814, 0.111057, 0.00155805, -0.111056, 0.993813]
transform=[-32.5317, -1.03079, 1.90774]

Is "transform" represent IMU depth extrinsic?

[zhonghong322]

Which device are you using?

[hplegend]

Which device are you using?

Femto Bolt.
Futhermore, When I set S2W align, pipeline.get_camera_param() return params show transform matrix is a identity matrix, is reasonable?

[john-maidbot]

hello, I have the same question for the Femto sensor. the ros2 node claims the transform is identity

[orbbec_camera_node-1] [INFO] [1734238445.968262807] [camera.ob_camera_node]: Publishing static transform from gyro to depth
[orbbec_camera_node-1] [INFO] [1734238445.968292746] [camera.ob_camera_node]: Translation 0, 0, 0
[orbbec_camera_node-1] [INFO] [1734238445.968299676] [camera.ob_camera_node]: Rotation 0, 0, 0, 1

are the imu and the depth sensor so closely aligned on the devices that this is reasonable? or does this simply mean that the extrinsic is unknown to the device or unset?

I am on the tip of the ros2 main branch (98a1cafd9beec33d6433e82c8e9af5ea0307e9de)

Thank you for your time.

[john-maidbot]

I think I’ve Found the answer
“The origin [0,0,0] of the gyroscope is the same as the origin of the depth camera. The origin of the accelerometer aligns with its physical location”
https://www.orbbec.com/documentation/femto-bolt-coordinate-systems/#:~:text=The%20%5Bx%2Cy%5D%20coordinates,left%20pixel%20in%20the%20image.

to clarify, this would suggest the translation is identity, but the rotation is not.

[hplegend]

I think I’ve Found the answer “The origin [0,0,0] of the gyroscope is the same as the origin of the depth camera. The origin of the accelerometer aligns with its physical location” https://www.orbbec.com/documentation/femto-bolt-coordinate-systems/#:~:text=The%20%5Bx%2Cy%5D%20coordinates,left%20pixel%20in%20the%20image.

to clarify, this would suggest the translation is identity, but the rotation is not.

are you using Orbbec for VI-SLAM?

[john-maidbot]

Yes, currently i am experimenting with passing the color, depth, and imu data into rtabmap

[hplegend]

Yes, currently i am experimenting with passing the color, depth, and imu data into rtabmap

I have been try my effort using orbbec for VIO slam, and I use orb-slam3 for the testing. However, the data captured from orbbec failure do VIO slam. If you successfully doing that, can you share you experience?

[john-maidbot]

I have not used the femto or femto bolt sensor with orbslam3. But I can share how I am using rtabmap.

First, I am using the ros2 wrapper to handle the orbbec sensor (https://github.com/orbbec/OrbbecSDK_ROS2). Which is really nice because it handles all the sdk usage. I just have to modify the launch file for my camera of choice with the camera settings I want.

Second, I am following this rtabmap example (https://github.com/introlab/rtabmap_ros/blob/ros2/rtabmap_examples/launch/k4a.launch.py), which I have adapted to work with the orbbec femto (the only change is updating the topic names and replacing the k4a sdk node with the orbbec node). This launch configuration applies a madgwick filter to the imu data, passes the filtered imu orientation estimate into an rgbd odometry node, and then passes the rgbd odometry estimate into the mapping node. It works well as long as I do not move too quickly (since the orbbec femto and femto bolt use rolling shutter rgb cameras, which dont handle fast motion well).

Lastly, you must adjust the imu extrinsics. since (as I stated above), I suspect the orbbec sdk does not provide the correct extrinsic (or the device does not have the right extrinsic for some reason?) between the imu and the depth camera. before doing this, I could not use rtabmap with rgbd + imu input because the imu offset was incorrect and the moment i started to move the algorithm diverged. So make 100% sure you are giving the correct extrinsics to your slam software. I went so far as to plot the imu data over time, move the camera and confirm to myself that the imu data matched my expectations. In my case, i ended up adding a function that alters the imu to depth transform before it gets sent to rtabmap.

Side note, try rgbd slam first (no imu) to limit the number of things that can go wrong (orbslam3 supports this: https://github.com/UZ-SLAMLab/ORB_SLAM3/tree/master/Examples/RGB-D). Then once you have that working, try rgbd + imu.

Please let me know if any of this needs more clarification 🙂