Using the SideBySide flag with an overlay causes eye texture behavior issues.

[Hokage3211]

So I am attempting to make a 3D overlay application, and it's clear steam planned something like this, as they implemented the SideBySideParallel flag. But the behavior is all over the place and erratic in terms of the texture for each eye being updated in time. When using it in the unity editor and pressing play in unity, only the right eye has rendering problems such as jittering/tearing/artifacting, and I've confirmed it has to be something deeper and not my code, as I've swapped cameras, swapped textures per eye, swapped positions and methods, and the right eye always has problems, while the left eye remains relatively stable with only occasional jitters, and I've seen it work almost flawlessly so it's not that the program is simply too slow, the profiler says I am running at about 250 fps in terms of cpu timings.

Another issue comes into effect when making a build of it, the behavior changes completely, it feels like the program isn't getting enough resources to render or something, as both eyes become extremely jittery, and what's even more strange is it performs slightly better when the window is not focused, and can even crash when the window is left as the top window on the computer. I will provide example builds below, but I want to explain a tiny bit more.

I'm using dantman's pull request code (seen here: #336 ) along with some minor changes of my own that make it easier to use and changes to make the overlay easy to move around in 3D space, to match the user's head position. I have a build using his code, to make it able to launch and not close other VR applications, but I have another build that is it as a registered steamVR program, using the normal code paths with unity XR enabled and such. Another strange behavior is that only when using the editor and pressing play while it's not using standalone mode, it sometimes performs better than usual, but wasn't reliable and sometimes also performed just as bad, often.

To also prove it's not a camera/sync issue in my program, when setting it to side-by-side crossed overlay and changing nothing else, it still has issues on only the right eye in the editor, showing it is something steamVR/the plugin is doing.

Build samples here: https://www.mediafire.com/file/hkzu9o1mm8gfpep/OverlayTesting.zip/file
two folders are inside, one "non-overlay" build which is the program registered as a steamVR game, showing the same issues occur even if it's not using dantman's code, and a testing build that is one that can run as an overlay application, showing the intended mode of the overlay where it runs as an overlay for any application/environment.

The distortion/controllers may be a little wonky, as it's hard-coded right now to my 70mm ipd, but the distortion is a separate issue to my jittering. The controllers outline are also an attempt to make normal controllers show in front of objects, since it's actually close to your eyes.

I hope all my information helps, and let me know if you need anything else, I listed everything I could think of in terms of behavior, to try and narrow it down and make it easy for you all.

[Hokage3211]

Oh yes, and this was requested to be opened by Joe Ludwig as I posted on another topic and he recommended I separate it into a new issue.

[Hokage3211]

Update with some more research:
I ended up adding a FPS counter to prove the program itself wasn't slow/lagging, but then I had the idea that it may be sending too much information to the overlay, causing issues, so I attempted to cap the framerate to 90, which improved it drastically but it seems it still only performs better when you minimize the window, which gives semi-perfect results but doesn't reproduce what I see in the editor pressing play, which is the right eye has more artifacts/update issues, and the left eye has the normal occasional jumps. Here is an updated version, https://www.mediafire.com/file/ss0hsrl5jzlu20d/TestingBuild.zip/file

Is there any way to hook into steamVR's rendering loop and regulate the overlay update framerate that way?

[Hokage3211]

Again with some more research, I streamlined the render process a bit, which helped, but it still has random artifacts in-editor, and when built and as a window that's focused, it goes crazy laggy, but then when minimized it performs actually almost perfect, but my tracking seems to be consistently a frame behind, as the whole thing sways just a bit in the direction of my motion, meaning the screen is not updating till one frame late. Any ideas or other known problems? I know unity is dropping support for openVR and valve will be picking it up, will they be making a better/more functional version of the Overlay class?