[Bug] WebGL Rendering Issues on Mobile Devices and Proposed Solutions

[sedghi]

Bug description

The WebGL rendering pipeline unfortunately has issues on several mobile devices, as previously reported in #3027 and cornerstonejs/cornerstone3D#1759 and cornerstonejs/cornerstone3D#1183 and OHIF/Viewers#4286 and other communications.

The main issue seems to be the hardware, and more recently, as discovered by @slak44, some software reports support for WebGL extensions but cannot perform linear filtering.

• For example, iOS devices cannot render float linear textures (see here).
• Different Pixel phones have various GPUs and different behvaiour. For instance, as reported by @daker, the Pixel 7A cannot render norm16 linear textures

We are facing more patches that fix some issues but create others. For instance, forcing half float when linear float is unavailable (which iOS lacks). The complexity of this code section is increasing, with multiple variables like norm16, norm16 linear filterability, half float enabled, halfFloat is exact, and float_linear extension availability and fallbacks for it if not. This suggests we need a better way to manage the pipeline for WebGL data types and internal formats.

I'm opening this issue to discuss.

I have created a simple HTML page to test various webgl extensions with canvas renderings of sample data to ensure the host machine can handle it. You can run it at https://debug.ohif.org (source code here )

Proposal

To permanently resolve this issue, we should test all rendering pipelines by priority and choose appropriate fallbacks for each in case of failure, I think ...

So my question is how much do you agree, and if i want to implement a fix (which will be a breaking change most likely):

1. Should consumers (like cornerstone3D) run these tests and set some flags within vtk.js, if so where they should be set? then later vtk.js pick the rendering specs based on the dataType and this prior info?
2. Or are you open to me adding an initializer inside renderWindow, possibly within getGLInformations, similar to other checks? My performance tests indicate that these six additional calls to test small canvas will add 30 ms, which is significant but happens only once. We can certainly do more profiling or optimize the code.

Steps to reproduce

Run vtk.js example in different phones in BrowserStack

Running https://kitware.github.io/vtk-js/examples/PiecewiseGaussianWidget.html

Huawei P30 (Failed to load OES_texture_float_linear. Texture filtering is not available for *32F internal formats.)

iPhone 14

Detailed Behavior

Inconsistencies in hardware and software implementations from manufacturers cause rendering issues in our pipeline.

Expected Behavior

Handle each phone appropriately according to its GPU.

Environment

• vtk.js version: latest
• Browsers: chrome , firefox
• OS: android, ios