App crashes after few minutes

[evia100]

Hi,

I used this repo to save the sensors' streams to my pc using the python script.
I wanted to get those streams (PV,VLC sensors,depth AHAT) and I have noticed 2 problems.

1. After few minutes (2 or 3 minutes) the app crashes. maybe due to RAM overflow.
2. I modified the python script a little bit and used opencv imwrite to write the different incoming images to my PC. there is a big lag and the writing of the images happens really slow (not at 30 fps for the PV or about 15-20 for the VLC as I expected).

basically, the purpose is to get the captured data from all those streams so I could post process the data. I don't need it to be shown in real time.
how could I do it?

Thank in advance,
Eviatar.

[cgsaxner]

Hi Eviatar,

if you want to process the sensor data from the HoloLens offline, I would recommend you to use the Stream Recorder App from the official HoloLens2ForCV repo from Microsoft.
This recorder App allows you to record the sensor streams directly on the HoloLens, and the repo also provides utility scripts for downloading the frames from the device to PC and some examples for postprocessing. (Note, however, that in my experience, the fps you expect cannot be reached if you record several sensor streams at once.)

The code in this repository is adapted from the recorder App and intended for online processing of the data.
Hope this can help!

[evia100]

Hi Christina, thank you for your comment. I firstly tried to use the stream recorder app as you suggested but I came across different problems. One problem was that the frames from different VLC sensors are not synchronized properly,the second problem is that after about one minute of recording the HoloLens get stuck and the app works really slow (and sometimes just crashes). Thats why I tried to write the frames back to the PC and maybe to reduce the IO process from the HoloLens to my PC.

I wish to hear from your experience, what fps to expect (if I want the PV and depth and maybe VLC sensors also) and how long one record can be (before a crash or someting like that)?

Thank a lot, it is very helpful.
Eviatar.

[cgsaxner]

Hi Eviatar,
oh, I see! In fact, my experience with the recorder was also mixed, but I only experienced crashes of the App when the local storage of the HoloLens was full. This can happen quite quickly when you record all sensor streams at once, in particular, the PV camera produces a lot of data. But I was able to record for several minutes usually, up to 4 or 5.

As for the issue of the VLC frames not being synchronized, I'm also facing this problem and have not found a solution. I did some experiments there and found that the sensors themselves are not synchronized, so already when you call GetNextBuffer on the IResearchModeSensor, you receive unsynchronized frames, and as far as I know, this is the only way of grabbing frames from the sensors. I could not find any parameters or access to the low-level hardware trigger with which you could modify that.

As for the framerates, I only did measurements with the VLC frames. The framerate depends on how many research mode sensors you grab - if you only record 2 VLC frames for example, you can get 30 fps. But even this is not constant - for example, when you move your hands in front of the HoloLens and the on-board hand tracking is triggered, the fps might drop to as low as 1-2 fps for a couple of frames. If you grab more than two VLC streams or also grab depth, the maximal fps you can get from VLC goes down to around 15 fps (again, not constant). The PV seems to be unaffected by this at large, but I did not do any experiments / measurements there.

All of this seems to be independent of what you do with the frames. So whether you stream them to PC, write them to the internal storage, or process them directly on the device, if you want to do some actual CV with the HoloLens, you are facing a lot of hurdles there :/

[raashidkhan1]

Hi,
I'm using the plugin to getLongDepthThrow images, by following RMCameraReader.cpp SaveDepth from the samples, I have been to trying to find the cause of crash at

std::shared_ptr<IResearchModeSensorDepthFrame> spDepthFrame(m_pDepthFrame, [](IResearchModeSensorDepthFrame* sf) { sf->Release(); });

runs for sometime but gets me access violation on release call, any idea how to fix that, below is my implementation.

`BYTE* __stdcall HL2Stream::GetLongThrowDepthData(int* dataSize)
{
OutputDebugStringW(L"Getting long throw.\n");

OutputDebugStringW(L"sensor and depth frame acquiring\n");

std::vector<BYTE> depthPgmData;
BYTE* data = nullptr;

OutputDebugStringW(L"sensor and depth frame camerareader\n");
IResearchModeSensorDepthFrame* m_pDepthFrame = nullptr;
IResearchModeSensorFrame* pSensorFrame = nullptr;

if (m_cameraReader->m_pSensorFrame)
{
	pSensorFrame = m_cameraReader->m_pSensorFrame;


	// Check if the frame is a depth frame
	HRESULT hr = pSensorFrame->QueryInterface(IID_PPV_ARGS(&m_pDepthFrame));

	
	OutputDebugStringW(L"sensor and depth frame lock\n");
	if (m_pDepthFrame && SUCCEEDED(hr))
	{
		std::shared_ptr<IResearchModeSensorDepthFrame> spDepthFrame(m_pDepthFrame, [](IResearchModeSensorDepthFrame* sf) { sf->Release(); });

		if (spDepthFrame) {
			depthPgmData = m_cameraReader->GetLongThrowDepthData(pSensorFrame, spDepthFrame);

			OutputDebugStringW(L"sensor and depth frame acquired\n");

			// Set the dataSize variable
			*dataSize = static_cast<int>(depthPgmData.size());

			data = new BYTE[*dataSize];

			std::copy(depthPgmData.begin(), depthPgmData.end(), data);

			OutputDebugStringW(L"got long throw.\n");
			size_t vectorSize = depthPgmData.size();

			// Convert size to a wide string and format the message
			std::wostringstream woss;
			woss << L"Vector size: " << vectorSize << L"\n";

			// Output the message
			OutputDebugStringW(L"depth data size: ");
			OutputDebugStringW(woss.str().c_str());
		}
	}
}
if (data == nullptr) {
	OutputDebugStringW(L"got no sensor frame ");
}
return data;

}`