Home
The [Tecla Framework] (https://github.com/idrc/tecla-as) makes use of the Android Accessibility Service to provide universal switch access to Android devices. The Tecla Framework is compatible with devices running Android Jelly Bean (API Level 16) and above.
The [Tecla Framework] (https://github.com/idrc/tecla-as) represents a paradigm shift in the way Tecla resolves switch access for Android devices. Starting from ICS (Android API Level 14), it is possible to perform programmatic navigation of UI elements through the [AccessibilityNodeInfo class] (http://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo.html). Furthermore, Android Jelly Bean (API Level 16) incorporated the ability to control each of those nodes programmatically. This has the potential to overcome some of the limitations with the keyboard injection method for control employed in the original [Tecla Access app] (https://github.com/idrc/TeclaAccess). Limitations such as the inability to access modal windows, options menus and the notifications drop-down bar, should all be addressable using the Tecla Framework. In particular, the Tecla Framework should overcome limitations currently preventing effective switch access to emerging home automation technologies and appliances.
The proposed user experience paradigm for the Tecla Framework is based on a semi-transparent overlay similar to heads-up displays (HUD) available in fighter aircraft (see image below). The HUD would render sizeable switch controls for interaction with the device. It is thus assumed this represents and advantage over the discrete navigation bottom row introduced with the original Tecla Access project for the following reasons:
- A HUD increases the space available for the arrangement of switch controls
- A HUD provides for the possibility of a more intuitively located arrangement of directional controls.
- The HUD aims towards a look and feel that unifies switch controls and device navigation into a single integrated user experience.
- The HUD provides limitless visual interaction possibilities that are not tied to the keyboard interaction paradigm provided with the original Tecla Access project.
The image below shows a mockup of a potential HUD overlay for the Tecla Framework.
In addition to the proposed user experience, the proposed transition from a SPP- to a HID-based interaction of the Shield with the Android device, offers an additional opportunity to access a wide range of input commands that can be used to control many more aspects of the OS and running apps than currently possible. However, the demand for efficient access to the myriad of input commands that must be accessible via a single-switch increases significantly. The proposed HUD design can also be used to provide Shield state and other information to the user so s/he can navigate the device efficiently via HID commands (keyboard, mouse and/or joystick).
After failed experiments with the development of a permeable system overlay and external input device inspection through the soft IME, it has become clear that the AccessibilityService is the next-best alternative for overcoming current limitations in the Tecla Access architecture. Enhancements exposed by the Accessibility Service include both enhanced highlighting and programmatic control of the Android platform.
Starting with API Level 14 (Ice Cream Sandwich) it is possible to use the AccessibilityService class to obtain information on the state and components of the Android user interface (UI). This information can be used to offer enhanced highlighting of the focused UI elements that receive input events (including Tecla's) at any given time.
The screenshot above depicts the application of the Accessibility Service framework to the development of a semi-transparent overlay used to draw bounding boxes on top of nodes 'focused' to receive user input. Two bounding boxes (white over dark) are drawn in order to ensure high contrast regardless of the background color.
Starting from API Level 16 (Jelly Bean) the AccessibilityService class exposes a means to control the user interface through the AccessibilityNodeInfo class. This will be the primary means for control of the Android platform, through the proposed HUD, within the Tecla Access framework.
One of the most interesting potential enhancements that our transition to the HID-based architecture provides is access to switch-based pointer emulation. Because pointer-based access is closer to the typical mobile device interaction paradigm than keyboard-based access, pointer-based interaction can provide the means to overcome some current limitations such as the inability to emulate touch gestures that do not have keyboard equivalents (e.g., swiping) or the unresponsivenes of most games to keyboard commands.
Our experiments with pointer-based navigation are documented on the Pointer HUD Experiments page.