Tracking Lost Service is not behaving correctly

[edgar-rodrigo-santos]

Describe the bug

The tracking lost service (https://github.com/MicrosoftDocs/mixed-reality/blob/docs/mrtk-unity/mrtk2/features/extensions/lost-tracking-service.md) doesn't work correctly and behaves very differently inside the app in comparison with what happens at an OS level.

At first, the "Finding your space" didn't even appear at all. That issue was already reported so I bumped into this: #11407. This change improved it, but it is still not working correctly and it is still not behaving like the OS does:

• OS doesn’t show the “Finding your space” as easily in badly mapped areas like MRTK does. We get too many "false positives" inside the app.
• The OS eventually recovers from that state. MRTK does not, unless we go back to a perfectly mapped area.

At the moment, it is impossible for our app to make a distinction between "completely covering the sensors" and "incomplete spatial information that we can live with".
In the OS this behaves correctly so we are thinking this could be a bug somewhere.

To reproduce

It is not straightforward to explain how to reproduce but I did reproduce it easily.

1. Take a walk around a space while in the OS (not running any app) and enter a badly mapped area (it doesn't even need to be very bad, just a bit incomplete). Make sure the OS got the "Finding your space" message and recovered from it (or that it didn't appear at all)
2. Do a similar test with an MRTK2 app with the Tracking Lost Service ON. Notice how the "Finding your space" message appears much more easily and it doesn't go way if we stay in the same place.

Note that covering the sensors has the same behavior in both OS and MRTK2, so it is not a good way to reproduce this issue.

Expected behavior

The expected behavior for us is exactly what the OS does. (no "false positives", and the message going away eventually in badly mapped areas)

Your setup

• Unity 2021.3.17f1
• MRTK 2.8.2.0 + Fixes #11340 - Splash image is not set when tracking is lost in hololens #11407

Target platform

• HoloLens 2

[AMollis]

@edgar-rodrigo-santos , are you using OpenXR? If so, OpenXR origin anchoring works a little differently than the older WMR strategies (i.e. what the Shell uses).

If you are using OpenXR, to correctly handle tracking lost, you need to put the application's tracking mode into an "Unbounded Reference Space". The easiest way to do this in MRTK2 is to add the EyeLevelSceneOrigin component to your MRTK2 rig's XR origin component.

Please try this, and reply back.

‼️ MRTK2 support is limited, and we recommend developers start looking at MRTK3 (mixedrealitytoolkit.org). If this is a new app that you are working on, we highly recommend using MRTK3 instead.

[edgar-rodrigo-santos]

@AMollis Thank you for your reply.
Yes, I'm using OpenXR.
I just tried adding the EyeLevelSceneOrigin and the behavior seems to be even worse.

If I cover up the sensors the tracking state doesn't change at all (even if I cover the sensors for a long time).
When I uncover them, it goes to limited for a while.
I also tried going into a completely dark room and the tracking state didn't change either.

[AMollis]

@edgar-rodrigo-santos I assume you are using AR Foundation 4.2, along with AR Subsystems 4.2?

@edgar-rodrigo-santos I couldn't reproduce the issue you are having, even after clearing my HL2's maps. However, I saw a slightly different problem. Tracking would never become completely lost, just limited. The service doesn't show the tracking lost message when in a limited state, see LostTrackingService::Update:

public override void Update()
{
    using (UpdatePerfMarker.Auto())
    {
        XRSessionSubsystem sessionSubsystem = SessionSubsystem;
        if (sessionSubsystem == null)
        {
            return;
        }

        if (sessionSubsystem.trackingState == lastTrackingState && sessionSubsystem.notTrackingReason == lastNotTrackingReason)
        {
            return;
        }

        base.Update();

        // This combination of states is from the Windows XR Plugin docs, describing the combination when positional tracking is inhibited.
        if (sessionSubsystem.trackingState == UnityEngine.XR.ARSubsystems.TrackingState.None && sessionSubsystem.notTrackingReason == NotTrackingReason.Relocalizing)
        {
            SetTrackingLost(true);
        }
        else
        {
            SetTrackingLost(false);
        }

        lastTrackingState = sessionSubsystem.trackingState;
        lastNotTrackingReason = sessionSubsystem.notTrackingReason;
    }
}

Mainly this line:

sessionSubsystem.trackingState == UnityEngine.XR.ARSubsystems.TrackingState.None

The service only indicates a loss in tracking when the state gets to 'none', which I had a hard time achieving. I had to change this line to the following, so I could see the lost tracking message:

(sessionSubsystem.trackingState == UnityEngine.XR.ARSubsystems.TrackingState.None || sessionSubsystem.trackingState == UnityEngine.XR.ARSubsystems.TrackingState.Limited) 

After the device reobtained tracking, the "lost tracking" message disappeared.

I still suspect that your issue is because of a missing or misconfigured EyeLevelSceneOrigin component. Here's my sample XRRig setup, that worked for me. Perhaps compare this with what you have, please notice the EyeLevelSceneOrigin on the root XRRig game object:

Another thing to check, ensure you see this message in your Unity log. If you don't see this, then your app doesn't have a XRInputSubsystem running or you are missing the EyeLevelSceneOrigin:

EyeLevelSceneOrigin: TrySetTrackingOriginMode to Unbounded

Also make sure you don't see this immediately after the first message:

EyeLevelSceneOrigin: Failed to set tracking origin to Unbounded

[edgar-rodrigo-santos]

@AMollis Thank you so much for the detailed reply.

After putting everything together like you described, it started behaving much better.

Even though we have ARFoundation in the project, we never had to use the ARSession and ARSessionOrigin components (neither did most if not all mrtk samples, at least in mrtk2).

We will do further testing to confirm nothing else was affected by these changes.

Thank you.

[AMollis]

@edgar-rodrigo-santos , thank you for the response. I apologize for the lack of documentation here. The Tracking Lost Service is actually the only MRTK2 component that relies on the AR Session. 😮 There's a gap in my history as to why this change was made initially

• Update LostTrackingService for Unity 2020 and XRSDK/WMR #8282

However, the WorldManager APIs were removed in Unity 2021, so this the why these is a need for AR Session. Unfortunately, I think it was an oversight that the Tracking Lost Service documentation wasn't updated.

There is some additional documentation on setting up AR Foundation and AR Session with MRTK2, but this experimental documentation was for iOS and Android. https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/supported-devices/using-ar-foundation?view=mrtkunity-2022-05

I apologize for the poor documentation here. Again, I recommend you investigate using MRTK3. Moving from MRTK2 to MRTK3 is large effort, so it might not be feasible. However, if this is a new application, I strongly recommend MRTK3.

Given you found a working solution, I'll be closing this issue. If you have further problems, please reach out, or open a new issue.