How to get origin SpatialCoordinateSystem using MRTK + OpenXR

[cgsaxner]

Overview

This documentation states that MRTK provides already marshalled types for getting the SpatialCoordinateSystem through the WindowsMixedRealityUtilities class. However, this class does not seem to contain those members.

With Windows XRSDK, I obtained the origin SpatialCoordinateSystem like this:

      IntPtr WorldOriginPtr =
            UnityEngine.XR.WindowsMR.WindowsMREnvironment.OriginSpatialCoordinateSystem;

        SpatialCoordinateSystem unityWorldOrigin = System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(
            WorldOriginPtr) as SpatialCoordinateSystem;

But with the new OpenXR SDK, this is not working anymore. I tried to access the SpatialCoordinateSystem using IWindowsMixedRealityUtilitiesProvider interface:

IntPtr WorldOriginPtrMRTK = 
     MRTK.WindowsMixedReality.WindowsMixedRealityUtilities.UtilitiesProvider.ISpatialCoordinateSystemPtr;

but this throws an exception.

Expected behavior

The underlying SpatialCoordinateSystem can be accessed like

SpatialCoordinateSystem unityWorldOrigin = 
     Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityUtilities.SpatialCoordinateSystem

Actual behavior

Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityUtilities does not contain a definition for SpatialCoordinateSystem, and other methods do not work with OpenXR SDK.

Unity editor version

Unity 2020.3.13f1

Mixed Reality Toolkit release version

MRTK 2.7.2

[keveleigh]

SpatialCoordinateSystem as a type doesn't really exist in OpenXR. What are you looking to use it for?

If you're looking to locate a second spatial coordinate system relative to it, you should be able to use the Mixed Reality OpenXR Plugin's SpatialGraphNode API, like so:

MixedRealityToolkit-Unity/Assets/MRTK/Providers/Experimental/WindowsSceneUnderstanding/WindowsSceneUnderstandingObserver.cs

Lines 796 to 800 in e140bf2

	SpatialGraphNode node = SpatialGraphNode.FromStaticNodeId(sceneOriginId);
	if (node.TryLocate(FrameTime.OnUpdate, out Pose pose))
	{
	result = Matrix4x4.TRS(pose.position, pose.rotation, Vector3.one).ToSystemNumerics();
	}

You'll need the spatial graph ID for the second object, but depending on what you're doing that should be accessible.

[cgsaxner]

Yes, I use the SpatialCoordinateSystem to locate my device (HoloLens 2) using, e.g., SpatialLocator.TryLocateAtTimestamp, and for locating frames, e.g. using SpatialCoordinateSystem.TryGetTransformTo.

After playing around a bit, I managed to get a SpatialCoordinateSystem in Unity with OpenXR using the scene understanding SDK like this:

using System.Threading.Tasks;
#if ENABLE_WINMD_SUPPORT
using Windows.Perception.Spatial;
using Windows.Perception.Spatial.Preview;
using Microsoft.MixedReality.SceneUnderstanding;
using Microsoft.MixedReality.OpenXR;
#endif

void Start()
{
#if ENABLE_WINMD_SUPPORT
    // gain access to the scene object
    SceneObserverAccessStatus accessStatus = Task.Run(RequestAccess).GetAwaiter().GetResult();
    if (accessStatus == SceneObserverAccessStatus.Allowed)
    {
        Scene scene = Task.Run(GetSceneAsync).GetAwaiter().GetResult();
        SpatialCoordinateSystem sceneCoordinateSystem = 
                   SpatialGraphInteropPreview.CreateCoordinateSystemForNode(scene.OriginSpatialGraphNodeId);
    }
    else
    {
        Debug.WriteLine("Something went wrong getting scene observer access!");
    }
#endif
}

private async Task<SceneObserverAccessStatus> RequestAccess()
{
    return await SceneObserver.RequestAccessAsync();
}

private async Task<Scene> GetSceneAsync()
{
    Scene scene = null;

    // Create Query settings for the scene update
    SceneQuerySettings querySettings;
    querySettings.EnableSceneObjectQuads = true;                                       // Requests that the scene updates quads.
    querySettings.EnableSceneObjectMeshes = true;                                      // Requests that the scene updates watertight mesh data.
    querySettings.EnableOnlyObservedSceneObjects = false;                              // Do not explicitly turn off quad inference.
    querySettings.EnableWorldMesh = true;                                              // Requests a static version of the spatial mapping mesh.
    querySettings.RequestedMeshLevelOfDetail = SceneMeshLevelOfDetail.Fine;            // Requests the finest LOD of the static spatial mapping mesh.

    // Initialize a new Scene
    scene = await SceneObserver.ComputeAsync(querySettings, 10.0f);

    return scene;
}

However, I could not confirm yet that sceneCoordinateSystem is actually equal to the Unity origin coordinate system. The documentation of the Scene understanding SDK suggests that some more conversions are needed, but I can't get this sample code to work, since I can't find the HolograhicFrameData datatype in any of the namespaces. Again, MRTK documentation suggests that it should live in Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityUtilities, but the definition seems not to exist there.

[keveleigh]

However, I could not confirm yet that sceneCoordinateSystem is actually equal to the Unity origin coordinate system.

Yeah, when running on OpenXR, I'm not sure there's a guarantee that these types will still be valid. OpenXR uses different types for locatability, which I wrote about at #10082 (comment).

I can't find the HolographicFrameData datatype in any of the namespaces

HolographicFrameNativeData is provided as a copyable snippet in the non-MRTK docs but is only relevant for legacy XR. If you're running Unity 2020+, that struct is no longer exposed anywhere, either in MRTK or Unity.
CurrentHolographicFrame as a HolographicFrame type should be accessible on the MRTK utilities class when using UWP.

[cgsaxner]

CurrentHolographicFrame as a HolographicFrame type should be accessible on the MRTK utilities class when using UWP.

If you are referring to the properties described here, unfortunately, no. For me, the WindowsMixedRealityUtilities class only exposes the methods described in the docs, but none of the properties, except for WindowsMixedRealityUtilities.UtilitiesProvider, which I don't know how to use (or whether I should even use it). Again, I tried this

IntPtr framePtr = WindowsMixedRealityUtilities.UtilitiesProvider.IHolographicFramePtr;

which throws an ArgumentOutOfRangeException.

[keveleigh]

Okay, I have a better update for you! I recently found out that you can access the root SpatialCoordinateSystem via a helper function in the Mixed Reality OpenXR Plugin package:

using Microsoft.MixedReality.OpenXR;
using Windows.Perception.Spatial;
SpatialCoordinateSystem origin = PerceptionInterop.GetSceneCoordinateSystem(UnityEngine.Pose.identity) as SpatialCoordinateSystem;

The resulting Windows.Perception.Spatial.SpatialCoordinateSystem object is located at the given pose in the current Unity scene.

[cgsaxner]

Thanks a lot @keveleigh, this is exactly what I was looking for! Just tested it and it works great for my applicaiton.

Just for completeness, meanwhile, I came up with this, which also seems to work:

SpatialLocator locator = SpatialGraphInteropPreview.CreateLocatorForNode(sensorGuid)
Scene scene = Task.Run(GetSceneAsync).GetAwaiter().GetResult();
SpatialCoordinateSystem sceneOrigin =SpatialGraphInteropPreview.CreateCoordinateSystemForNode(scene.OriginSpatialGraphNodeId);
SpatialGraphNode node = SpatialGraphNode.FromStaticNodeId(scene.OriginSpatialGraphNodeId);
if (node.TryLocate(FrameTime.OnUpdate, out Pose pose))
{
    sceneToWorldMatrix = Matrix4x4.TRS(pose.position, pose.rotation, Vector3.one);
}
else
{
    Debug.WriteLine("Could not obtain the scene to world matrix.");
}

And then use the sceneOrigin as coordinate system, and sceneToWorldMatrix to transform poses in this coordinate system to the Unity world coordinate system. But of course, your method is much nicer and there are less pitfalls.

[NikolayMXR]

For someone who is trying to move scene understanding to the MRTK+openxr from old approach don't forget to ConvertRightHandedMatrix4x4ToLeftHanded on the root object after using SpatialGraphNode.FromStaticNodeId(scene.OriginSpatialGraphNodeId);

[keveleigh]

Perfect, thank you for following up! I'm going to close this issue down now as it seems we have a solution. Please reopen if I'm mistaken. Thanks!