How can I get world space ray from pv frame ?

[kaiwu119]

I want get a ray from pv frame pixel coordinats, first I use as follow to get a ray in pv camera space :
pvFrame->SensorStreamingCameraIntrinsics->MapImagePointToCameraUnitPlane(uv, &xy);
but it not work, when I deploy app to hololen, the app crashes directly(I only add one line, as bold code)

	DepthPvMapper::DepthPvMapper(HoloLensForCV::SensorFrame^ depthFrame, HoloLensForCV::SensorFrame^ pvFrame)
	{
		_imageToCameraMapping = createImageToCamMapping(depthFrame);
		**_pvImageToCameraMapping = createImageToCamMapping(pvFrame);**
		
	}
	
cv::Mat DepthPvMapper::createImageToCamMapping(HoloLensForCV::SensorFrame^ sensorFrame)
	{
		cv::Mat imageToCameraMapping = cv::Mat(
			sensorFrame->SoftwareBitmap->PixelHeight,
			sensorFrame->SoftwareBitmap->PixelWidth,
			CV_32FC2,
			cv::Scalar::all(0));
		for (int x = 0; x < sensorFrame->SoftwareBitmap->PixelWidth; ++x) {
			for (int y = 0; y < sensorFrame->SoftwareBitmap->PixelHeight; ++y) {
				Windows::Foundation::Point uv = { float(x), float(y) };
				Windows::Foundation::Point xy(0, 0);
				
				if (sensorFrame->SensorStreamingCameraIntrinsics->MapImagePointToCameraUnitPlane(uv, &xy)) {
					imageToCameraMapping.at<cv::Vec2f>(y, x) = cv::Vec2f(xy.X, xy.Y);
				}
			}
		}
		return imageToCameraMapping;
	}

I also try to get the inverse of the projection matrix of pv frame，this will also crash the program.
Can someone help me ？ Thanks in advance！

[emeraldy]

Hi,

I think the pv camera uses a different function for unprojection: there is a CoreCameraIntrinsics object in the SensorFrame class, which provides a method called UnprojectAtUnitDepth. Try this and you should get a ray in the pv camera space to start with. Since the camera is looking down at -z, the unit plane's z assumed by the function is -1. The documentation is here: https://docs.microsoft.com/en-us/uwp/api/windows.media.devices.core.cameraintrinsics.unprojectatunitdepth?view=winrt-19041#Windows_Media_Devices_Core_CameraIntrinsics_UnprojectAtUnitDepth_Windows_Foundation_Point_

[kaiwu119]

Thank you very much for your answer, this gave me a direction to try, thanks again.