Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement iOS VoiceOver support #18016

Open
wants to merge 31 commits into
base: master
Choose a base branch
from

Conversation

IsaMorphic
Copy link
Contributor

What does the pull request do?

This is a twin PR to #17704 that implements support for VoiceOver on iOS devices. Importantly, the UIAccessibility & UIAccessibilityContainer informal protocols are leveraged to create cohesion between Avalonia's automation system and that of the iOS ecosystem.

What is the current behavior?

Currently, Avalonia apps targeting the iOS platform do not expose user controls to the operating system's accessibility pipeline. This makes it impossible for blind and visually impaired users to use those apps with a screen reader or braille display.

What is the updated/expected behavior with this PR?

This PR enables users reliant on VoiceOver features to properly navigate Avalonia's view heirarchy using the appropriate, well-known shortcuts and gestures.

How was the solution implemented (if it's not obvious)?

The PR implements support for VoiceOver via Avalonia's AutomationPeer API by wrapping those instances in a subclass of UIAccessibilityElement that implements all the necessary formal and informal protocols to create a fully realized experience for low vision users.

Checklist

Sorry, something went wrong.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054393-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054403-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054407-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054449-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054459-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054504-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@IsaMorphic IsaMorphic marked this pull request as ready for review January 23, 2025 17:33
@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054514-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054524-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054546-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054548-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054550-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054706-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054801-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054816-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054856-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@IsaMorphic
Copy link
Contributor Author

I don't think my code caused those tests to fail... anyone know if there's a known CI issue that could have caused it?

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054884-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054946-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@maxkatz6
Copy link
Member

maxkatz6 commented Feb 16, 2025

I finally tested this branch on a simulator using Accessibility Inspector and have several questions:

  1. Voiceover part works great.
  2. Actions like "Activate" don't seem to work (for example, on Repeat Button). accessibilityActivate is not invoked as I tried to debug it. Other actions, like scroll on focus does work.
  3. For some elements, boundaries seem to be off. But for the majority of other elements (like simple buttons), it's correct.
  4. ControlCatalog has TabControl-based navigation, where only a single page is actually part of the visual tree. But it seems once tab was opened, it never gets removed from the voice-over tree. For example, if I open Buttons page and then go to AutoCompleteBox page, Accessibility Inspector still goes through buttons from the previous page.
Screenshot 2025-02-16 at 2 23 57 AM

A little rant: it's frustrating how sometimes difficult it is to get mobile workloads working after XCode update, and almost every time it's easier with vscode than rider.

@@ -221,7 +221,7 @@ protected override bool IsOffscreenCore()
IsOffscreenBehavior.FromClip => Owner.GetTransformedBounds() is not { } bounds ||
MathUtilities.IsZero(bounds.Clip.Width) ||
MathUtilities.IsZero(bounds.Clip.Height),
_ => !Owner.IsVisible,
_ => !Owner.IsEffectivelyVisible,
Copy link
Member

@maxkatz6 maxkatz6 Feb 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @grokys, is it fine to switch to Effectively Visible? Or do other backends handle inherited visibility on their own?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other backends handle this only in-so-far as they handle the hit testing in Avalonia code. This is in contrast to iOS which is the only target platform that performs the hit testing on the OS side. Therefore, I argue that this adaptation is necessary.

Comment on lines 56 to 71
foreach (AutomationPeer child in peer.GetChildren())
{
AutomationPeerWrapper? wrapper;
if (!_childrenMap.TryGetValue(child, out wrapper) &&
(child.GetName().Length > 0 || child.IsKeyboardFocusable()))
{
_childrenList.Add(child);
_childrenMap.Add(child, new(this, child));
}

wrapper?.UpdatePropertiesIfValid();
wrapper?.UpdateTraits();

UpdateChildren(child);
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not familiar with iOS accessibility, but shouldn't this backend map avalonia automation onto accessibilityContainer?

In other words, it should have a tree of peer wrappers, implementing IUIAccessibilityContainer, on wrapping children only on request. Instead of processing every element in the app into a single dictionary, updating it on each accessibility request. Otherwise, it needs to be carefully tested for performance and memory leaks, and potentially that problem with TabControl I mentioned above. At the moment, cleanup is only happening on GetAccessibilityElementAt request for a specific element, and if iOS doesn't call this method, no element is cleared, holding it in the memory.

By implementing IUIAccessibilityContainer on every node it would make it possible to properly handle accessibilityContainerType.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue I encountered trying to implement what you're describing is that iOS then doesn't traverse the tree when using touch exploration. In other words, it ends up just settling on the root element because, like you are saying, it doesn't traverse deeper due to the OS not requesting that traversal in the first place.

Unless I were to do some kind of implicit aggregation in the accessibilityContainer methods that does this traversal for the OS (which would again bring up the performance concerns), this is the only way I was able to get touch exploration to (mostly) work.

All this said, I can definitely double check the cleanup code to make sure that we don't get the weird TabControl behavior you're describing.

@IsaMorphic
Copy link
Contributor Author

IsaMorphic commented Feb 17, 2025

Thanks @maxkatz6 for the in-depth code review. Below are responses to your comments in numerical order:

  1. Thanks! :D
  2. I noticed this too but wasn't sure if it was just an issue on my end. I'll make sure that those overrides work on the native side of things.
    Edit: I've confirmed that this is just a bug with the iOS simulator and that the OS properly delegates this event on a physical iOS device.
  3. See 4.
  4. I've observed this behavior only occurs up until the OS requests a refresh of the accessibility tree. For example, if you navigate via touch exploration directly after the page switch, the outdated elements disappear as they should. So, I'll make sure to notify the OS of these view changes as soon as they occur.
    Edit: the refresh event never arrives on the Avalonia side 🤔 opened TabControl does not call AutomationPeer.RaiseChildrenChangedEvent #18227
    Edit 2: after some more digging it turns out that this wasn't the root issue. Instead it was regarding an implementation issue with IsEffectivelyVisible where orphaned trees would assume they are still visible (which doesn't make much sense in the grand scheme of things)

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054976-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0054989-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@IsaMorphic IsaMorphic requested a review from maxkatz6 February 18, 2025 15:05
@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0055033-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

@avaloniaui-bot
Copy link

You can test this PR using the following package version. 11.3.999-cibuild0055131-alpha. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants