Depth Function in RealSense SDK

[RaykiDan]

Hi, I'm a student and I'm currently learning computer vision especially in depth estimation topics.

I'm working on a project using RealSense D435i depth camera in python by using pyrealsense wrapper to run the librealsense in python language in Ubuntu 22.04 and I'm curious on how the RealSense SDK stereoscopic depth rendition works. I'm trying to find the depth function file to read the program itself. Can someone tell me which file/module is it? Thank you.

[MartyG-RealSense]

Hi @RaykiDan If you wish to know the basic principles of how the stereoscopic depth works then the link below has an introductory guide.

https://github.com/realsenseai/librealsense/blob/master/doc/depth-from-stereo.md

The RealSense data sheet document also provides the following explanation.

---

The RealSense D400 series depth camera uses stereo vision to calculate depth. The stereo vision implementation consists of a left imager, right imager, and an optional infrared projector. The infrared projector projects a non-visible static IR pattern to improve depth accuracy in scenes with low texture. The left and right imagers capture the scene and send imager data to the depth imaging (vision)
processor, which calculates depth values for each pixel in the image by correlating points on the left image to the right image and via the shift between a point on the Left image and the Right image. The depth pixel values are processed to generate a depth frame. Subsequent depth frames create a depth video stream.

[RaykiDan]

Thank you for the explanation @MartyG-RealSense.

Previously, I'm curious as to whether we can use each imager to produce the depth frame individually (creating not only 1 but 2 depth frame from each infrared frames), but looking at your answer it seems impossible to do so since we not only need 1 imager but also another one to correlates a point shifting. I guess that's

I've read the datasheet and found the following explanation:

The depth data generated by the D4 Vision Processor utilizes the left imager as the
reference for the stereo matching algorithm, resulting in a non-overlapped region in
the camera’s field of view.

There's also an image showing the Depth FoV to Depth Map Illustration, explaining the overlapping region between the imager is the valid depth:

I have several question regarding this:

1. Does this mean that what's showed in the depth frame is only what captured inside the valid depth?
2. Did the RealSense cam (for example in my case is D435i) automatically outputting the valid depth (overlapping region) only?
3. Is there any way to get the raw depth frame with invalid depth band? And can we flip it around and use the right imager as the reference instead?

[MartyG-RealSense]

The camera generates a depth frame using raw left and right infrared frames in the camera hardware. These are not the same as the frames accessed in the infrared stream. This is why it is possible to stream depth frames when the left and right infrared streams are not enabled.

You can access the sensors directly to control them using the SDK's Low-Level API but this feature is mostly undocumented and there is only one official example program - rs-data-collect - for accessing the Low-Level API. So implementing a solution for creating custom frames using that approach would be very difficult.

https://github.com/realsenseai/librealsense/blob/master/doc/api_arch.md#low-level-device-api

https://github.com/realsenseai/librealsense/tree/master/tools/data-collect

If you used the genlock hardware sync method of triggering frame capture using a trigger signal sent through a wire from an external signal generator then you could use genlock's 'burst' mode to generate more than one depth frame on each trigger.

Regarding valid depth, yes the image will not include depth that is within the invalid depth band image on the left side of the depth image. You can widen the depth field of view by using a camera model with a wider FOV than D435i, such as D455, or using two cameras and positioning them so that their fields of view overlap.

The invalid depth band area widens as the camera moves closer to an observed surface and becomes smaller as the camera is moved further away.

You cannot access the raw depth frame or change the reference. There has been a past example in #6311 (comment) of a RealSense user who chose to use OpenCV to generate their own depth frame from the left and right infrared frames instead of using the camera's depth frame.