[Feature]: Return depth, segmentation from camera sensor.

[vybhav-ibr]

What feature or enhancement are you proposing?

If a camera sensor is added through scene.add_sensor(**options) only RGB data can be read from this sensor(when the Rendering back-end is openGL). But when a camera is added through scene.add_camera depth, the cam.render() accepts arguments to render segmentation, surface normal and depth

It would be very convenient to have this feature, and it seems easy to implement

Potential Benefit

enables all the features of the scene.add_camera while maintaining a common API across sensors

[Milotrince]

You can add a depth camera with scene.add_sensor(gs.sensors.DepthCamera)

genesis-world/examples/sensors/lidar_teleop.py

Line 107 in a1363a7

sensor = scene.add_sensor(gs.sensors.DepthCamera(pattern=gs.sensors.DepthCameraPattern(), **sensor_kwargs))

The current camera vs. sensor camera interface is admittedly a bit confusing. I will update docs.

[vybhav-ibr]

I am aware of this, but doesn't it use compute raytracing ?. while the camera added through the scene.add_camera API uses rasterization(configurable to also use raytrcaing with madrona),
In my experience the DepthCamera sensor uses more VRAM

[Milotrince]

Yes this is true the computation is different for camera depth and DepthCamera. Returning depth and segmentation is a bit inconvenient in our current sensor pipeline assumes one dtype per sensor. @duburcqa do you have thoughts on how we want to handle this? Have camera sensor bypass the managed cache system or do we extend the managed cache system so multiple dtype returns is supported? (another pipeline rewrite 🥲 )

Also, camera sensor is already a special case and bypasses much of the standard pipeline and contains much duplication with vis.camera code, which I don't particularly like. I was thinking we could have sensor camera own a vis.camera (less wrappers and dup) and also enable the base sensor imperfection pipeline?

[vybhav-ibr]

How about having a four additional arguments for the camera options (rgb=True, ...) similar to the .render method of the pyrender camera. And the class CameraReturnType has 'rgb', 'depth', 'seg' as part of the tuple and these buffers are populated if the corresponding arg is true. The size would be determined at build time

Or, simply pack all the data as channels of one huge tensor(with one dtype, like uint8) and have helper methods to retrieve a convenient view of the data. Like camera_sensor.read_rgb() or camera_sensor.read().rgb()

I was thinking we could have sensor camera own a vis.camera (less wrappers and dup)

How would this work for nyx?