test: xr test automation #16204

simonedevit · 2025-02-23T10:31:33Z

The following PR proposes a solution to integrate WebXR automated tests.

Forum link: https://forum.babylonjs.com/t/webxr-test-automation/56736.

HOW TO CREATE A NEW TEST:

Start Babylon server with HTTPS (*)
npm run serve -w @tools/babylon-server
Start Babylon playground with HTTPS (*)
npm run serve -w @tools/playground
Wear your headset (**)
(headset) Go to https://{YOUR_LOCAL_IP}:1337/?local=true and accept authority certificates (required from HTTP calls by playground)
(headset) Go to https://{YOUR_LOCAL_IP}:1338/?local=true#F41V6N#2096 and accept authority certificates
(headset) Click on Settings icon > "RECORD XR SESSION"
As soon you enter in XR session, all actions (camera and controller movements) will be recorded
(headset) Enter in XR session, click on the sphere with your right controller (**) and the sphere will become red. Press on "A" button of right controller to save your recorded actions in a file
A file similar to F41V6N-2025-02-22_18_35_38.json will be saved in your device (for Oculus Quest 3 the folder is /sdcard/Download)
Enable "Developer options", connect your device with ADB and copy the above file
Past the file into packages/tools/tests/test/visualization/XRActions folder
Add the file name (inclusive of extension) in actions attribute of your current test in packages/tools/tests/test/visualization/config.json, e.g.:

{
    "title": "VR Test",
    "playgroundId": "#F41V6N#2096",
    "isWebXR": true,
    "actions": "F41V6N-2025-02-22_18_35_38.json"
}

Stop Babylon server and start it without HTTPS protocol
Start visualization tests
npm run test:visualization:ui
Play your test

(*) I have set https: true directly in common Webpack configuration (packages/dev/buildTools/dist/webpackTools.js) rather than use the launch scripts (.vscode/launch.json).

(**) The test was executed on an Oculus Quest 3.

bjsplat · 2025-02-23T10:32:21Z

Building or testing the sandbox has failed.

If the tests failed, results can be found here:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T10:32:23Z

Graph tools CI has failed you can find the test results at:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/TOOLS/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T10:33:05Z

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

bjsplat · 2025-02-23T10:33:05Z

Reviewer - this PR has made changes to one or more package.json files.

simonedevit · 2025-02-23T10:34:15Z

I'm sorry, I thought I had created the pull request on my fork.

bjsplat · 2025-02-23T10:35:34Z

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

bjsplat · 2025-02-23T10:35:35Z

Reviewer - this PR has made changes to one or more package.json files.

simonedevit · 2025-02-23T10:35:44Z

packages/dev/core/src/XR/webXRCamera.ts

-            this._xrSessionManager.referenceSpace = this._xrSessionManager.referenceSpace.getOffsetReferenceSpace(transform);
+            let isIwerEmulator = false;
+            // @ts-expect-error - augment Window interface
+            if (window && window.xrDevice){


Is this a robust condition to check if we are using Immersive Web Emulation Runtime?

simonedevit · 2025-02-23T10:39:31Z

packages/dev/core/src/XR/webXRCamera.ts

+            if (window && window.xrDevice){
+                isIwerEmulator = true;
+            }
+            const offsetMatrix = isIwerEmulator ? transform.matrix : transform;


IWER doesn't accept an XRRigidTransform object but a mat4 (gl-matrix) package. Maybe this is not compliant to WebXR Device API specifications (https://www.w3.org/TR/webxr/#xrreferencespace-interface).

simonedevit · 2025-02-23T10:40:40Z

packages/tools/playground/src/components/rendererComponent.tsx

@@ -358,6 +360,9 @@ export class RenderingComponent extends React.Component<IRenderingComponentProps
                globalObject.scene.then((s: Scene) => {
                    this._scene = s;
                    globalObject.scene = this._scene;
+                    if (Utilities.ReadBoolFromStore("record-xr-session", false)){
+                        this._addRecordActions();
+                    }
                });
            } else {
                this._scene = globalObject.scene as Scene;


No need to add this._addRecordActions because XR session is always async.

simonedevit · 2025-02-23T10:44:37Z

packages/tools/tests/test/playwright/visualizationPlaywright.utils.ts

@@ -113,12 +117,31 @@ export const evaluatePlaywrightVisTests = async (engineType = "webgl2", testFile
            page.on("console", logCallback);
            console.log("Running test: " + testCase.title, ". Meta: ", testCase.playgroundId || testCase.scriptToRun || testCase.sceneFilename);
            test.setTimeout(timeout);
+            let xrActions = null;
+            if (testCase.isWebXR){


Inject Immersive Web Emulation Runtime only if the current is an XR test case (this keep away potential regressions to other test cases for injecting a variable in the global object).

simonedevit · 2025-02-23T10:47:22Z

packages/tools/tests/test/playwright/visualizationPlaywright.utils.ts

+                xrSessionManager.onXRReady.add(() => {
+                    player = window.xrDevice.createActionPlayer(xrSessionManager.baseReferenceSpace, xrActions);
+                    // rewrite stop function otherwise IWER's stop function restore camera and controllers to initial position
+                    player.stop = () => { resolve(true) };


Resolve the promise once that the execution of recorded actions is over but don't stop the player because stopping the player means restore camera and controllers to initial position. Image scenario where you execute a lot of actions (e.g.: you move in the space) and you need to check/test the final view not the first one).

bjsplat · 2025-02-23T10:57:03Z

Building or testing the playground has failed.

If the tests failed, results can be found here:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T10:57:55Z

Building or testing the sandbox has failed.

If the tests failed, results can be found here:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T10:58:56Z

Graph tools CI has failed you can find the test results at:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/TOOLS/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T11:24:10Z

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

bjsplat · 2025-02-23T11:24:10Z

Reviewer - this PR has made changes to one or more package.json files.

bjsplat · 2025-02-23T11:45:29Z

Building or testing the sandbox has failed.

If the tests failed, results can be found here:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T11:46:00Z

Building or testing the playground has failed.

If the tests failed, results can be found here:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/16204/merge/testResults/

bjsplat · 2025-02-23T11:46:30Z

Graph tools CI has failed you can find the test results at:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/TOOLS/refs/pull/16204/merge/testResults/

RaananW · 2025-02-23T16:26:06Z

Hi and thanks for the PR!

Playwright allows us to install extensions on CI agents, so this is the solution I would pursue. I don't want to make any changes to the XR classes just for the sake of testing.

The changes made in this kind of PR should be minimal - XR should work the same as any other test just with an extra click on the XR button.

Also, as a side note - dual screen would make more sense, because we do have sometimes issues of things rendering incorrectly on the second render pass.

It's a future task of mine, but I still didn't dive too deep into that. There is also the fact that we are testing on browserstack (which might change the test scenario if they don't allow us to install plugins).

simonedevit · 2025-02-23T17:13:12Z

Thank you for your response!

I'm really glad to hear that this is one of your future tasks because i want to learn a lot from you! I am completely agree not wanting to modify XR classes just for testing purposes, my suggestion was just an assumption to get the job done.

Also, as a side note - dual screen would make more sense, because we do have sometimes issues of things rendering incorrectly on the second render pass.

Thank you for the tip! I hadn’t considered that rendering issues might occur on the second pass, definitely something to keep in mind.

Feel free to close the pull request if you think it's appropriate.

test: base structure for xr test automation

c15da0a

simonedevit closed this Feb 23, 2025

simonedevit reopened this Feb 23, 2025

simonedevit commented Feb 23, 2025

View reviewed changes

fix: remove debugger statement

f2d1b9b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: xr test automation #16204

test: xr test automation #16204

simonedevit commented Feb 23, 2025 •

edited

Loading

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

simonedevit commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

simonedevit Feb 23, 2025

simonedevit Feb 23, 2025

simonedevit Feb 23, 2025

simonedevit Feb 23, 2025

simonedevit Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

RaananW commented Feb 23, 2025

simonedevit commented Feb 23, 2025

test: xr test automation #16204

Are you sure you want to change the base?

test: xr test automation #16204

Conversation

simonedevit commented Feb 23, 2025 • edited Loading

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

simonedevit commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

simonedevit Feb 23, 2025

Choose a reason for hiding this comment

simonedevit Feb 23, 2025

Choose a reason for hiding this comment

simonedevit Feb 23, 2025

Choose a reason for hiding this comment

simonedevit Feb 23, 2025

Choose a reason for hiding this comment

simonedevit Feb 23, 2025

Choose a reason for hiding this comment

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

bjsplat commented Feb 23, 2025

RaananW commented Feb 23, 2025

simonedevit commented Feb 23, 2025

simonedevit commented Feb 23, 2025 •

edited

Loading