-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement iOS VoiceOver support #18016
base: master
Are you sure you want to change the base?
Conversation
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
I don't think my code caused those tests to fail... anyone know if there's a known CI issue that could have caused it? |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
@@ -221,7 +221,7 @@ protected override bool IsOffscreenCore() | |||
IsOffscreenBehavior.FromClip => Owner.GetTransformedBounds() is not { } bounds || | |||
MathUtilities.IsZero(bounds.Clip.Width) || | |||
MathUtilities.IsZero(bounds.Clip.Height), | |||
_ => !Owner.IsVisible, | |||
_ => !Owner.IsEffectivelyVisible, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @grokys, is it fine to switch to Effectively Visible? Or do other backends handle inherited visibility on their own?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Other backends handle this only in-so-far as they handle the hit testing in Avalonia code. This is in contrast to iOS which is the only target platform that performs the hit testing on the OS side. Therefore, I argue that this adaptation is necessary.
foreach (AutomationPeer child in peer.GetChildren()) | ||
{ | ||
AutomationPeerWrapper? wrapper; | ||
if (!_childrenMap.TryGetValue(child, out wrapper) && | ||
(child.GetName().Length > 0 || child.IsKeyboardFocusable())) | ||
{ | ||
_childrenList.Add(child); | ||
_childrenMap.Add(child, new(this, child)); | ||
} | ||
|
||
wrapper?.UpdatePropertiesIfValid(); | ||
wrapper?.UpdateTraits(); | ||
|
||
UpdateChildren(child); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not familiar with iOS accessibility, but shouldn't this backend map avalonia automation onto accessibilityContainer?
In other words, it should have a tree of peer wrappers, implementing IUIAccessibilityContainer, on wrapping children only on request. Instead of processing every element in the app into a single dictionary, updating it on each accessibility request. Otherwise, it needs to be carefully tested for performance and memory leaks, and potentially that problem with TabControl I mentioned above. At the moment, cleanup is only happening on GetAccessibilityElementAt request for a specific element, and if iOS doesn't call this method, no element is cleared, holding it in the memory.
By implementing IUIAccessibilityContainer on every node it would make it possible to properly handle accessibilityContainerType.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue I encountered trying to implement what you're describing is that iOS then doesn't traverse the tree when using touch exploration. In other words, it ends up just settling on the root element because, like you are saying, it doesn't traverse deeper due to the OS not requesting that traversal in the first place.
Unless I were to do some kind of implicit aggregation in the accessibilityContainer methods that does this traversal for the OS (which would again bring up the performance concerns), this is the only way I was able to get touch exploration to (mostly) work.
All this said, I can definitely double check the cleanup code to make sure that we don't get the weird TabControl behavior you're describing.
Thanks @maxkatz6 for the in-depth code review. Below are responses to your comments in numerical order:
|
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
You can test this PR using the following package version. |
What does the pull request do?
This is a twin PR to #17704 that implements support for VoiceOver on iOS devices. Importantly, the UIAccessibility & UIAccessibilityContainer informal protocols are leveraged to create cohesion between Avalonia's automation system and that of the iOS ecosystem.
What is the current behavior?
Currently, Avalonia apps targeting the iOS platform do not expose user controls to the operating system's accessibility pipeline. This makes it impossible for blind and visually impaired users to use those apps with a screen reader or braille display.
What is the updated/expected behavior with this PR?
This PR enables users reliant on VoiceOver features to properly navigate Avalonia's view heirarchy using the appropriate, well-known shortcuts and gestures.
How was the solution implemented (if it's not obvious)?
The PR implements support for VoiceOver via Avalonia's
AutomationPeer
API by wrapping those instances in a subclass ofUIAccessibilityElement
that implements all the necessary formal and informal protocols to create a fully realized experience for low vision users.Checklist