A call for auditable Virtual, Augmented and Mixed Reality

Richard Cloete, Chris Norval and Jatinder Singh
Compliant and Accountable Systems Group (http://www.compacctsys.net/)
University of Cambridge

To appear in VRST 2020 (https://vrst.acm.org/vrst2020/)

Overview

XR systems (Virtual, Augmented and Virtual Reality) are growing in prominence. With any new technology comes risk, and XR is no exception. Indeed, the novel interaction modalities and the immersive nature (manipulating/impeding senses) of XR presents some unique risks. This is particularly important given the adoption of the tech by more risk-averse sectors such as manufacturing, surgery, training, construction and the military, to name a few.

Auditability of XR systems can help in this regard. That is, capturing information about how XR systems are used, how they operate and how various components (internal and external) interact can help to unpack failures/harms be they due to usage, technical, administrative or social complications. This 'audit data' can also help identify failures before they occur, demonstrate that systems operate/perform in line with regulations and design specifications and more generally, can help towards making XR more compliant, transparent and safer -- properties that will undoubtedly affect the technologies adoption.

Survey background

We conducted a survey exploring the state of audit practices 'in the wild' for a range of emerging technologies and stakeholders.

Our research proposal for this work was reviewed was approved by our institution's ethical committee (for further information, see: www.research-integrity.admin.cam.ac.uk/research-ethics), and consent was obtained from all participants. All questions were optional and randomised, data was anonymised and participants gave their explicit consent for us to use their responses.

For this paper, we compiled our results based only on those participants that said that they work with XR (AR/VR/MR) technologies; we did this in order to make three broad and general points as elaborated in our paper.

Survey Questions

The following questions relate to your organisation and present questions about which technologies you work with, which sectors your organisation targets, etc.

Q1 - How many staff does your organisation employ? (select "1 employee" for solo developer)

1. 1 employee
2. 2-10
3. 11-50
4. 51-250
5. More than 250 employees

Q2. Which of the following technologies do you work with? Check all that apply.

1. Virtual Reality
2. Augmented Reality
3. Mixed Reality
4. Internet of Things
5. Artificial Intelligence / Machine Learning
6. Robotics
7. Other

Q3 - You’ve indicated that you work with technologies not listed above. Please elaborate in the text field below.

Q4 - What sectors do your organisation's products target? Check all that apply.

1. Arts/Culture
2. Civil Society (NGOs)
3. Construction
4. Education
5. Finance
6. Gaming
7. Government
8. Manufacturing
9. Medical
10. Mining/oil/gas
11. Military
12. Research
13. Security
14. Other

Q5 - Which other sectors do your organisations products target?

Q6- Please select the target customers for your products. Check all that apply.

1. Businesses (B2B)
2. Consumers (B2C)
3. Government/public sector/charities
4. Other

Q7 - You’ve indicated that your organisation uses other approaches to selling. Please elaborate in the text field below.

Q8 - At your current organisation, which tasks are you involved in? Check all that apply.

1. Developing software products (programming, database management, testing, architecting, etc.)
2. Analysing data (data science, user data analysis, etc.)
3. Management (leading a software team, overseeing the design of products, etc.)
4. Sales
5. Other

Q9 - You’ve indicated that you are involved in other/additional roles to the ones listed above. Please elaborate in the text field below.

Q10 - What sectors do your organisation's products target? Check all that apply.

1. Security
2. Other
3. Manufacturing
4. Medical
5. Construction
6. Education
7. Gaming
8. Finance
9. Military
10. Government
11. Civil Society (NGOs)
12. Arts/Culture
13. Research
14. Mining/oil/gas

The following questions relate to the recording of data during the design, development and testing stages of products.

Q11 - Which of the following types of data are recorded during design/development/testing?

1. Debug information (e.g. test records, performance information, debug logs, diagnostics, etc.)
2. Intermediate/generated data (e.g. interaction events, aggregated data, function outputs, etc.)
3. Meta information (versioning information, operating system, hardware, resources consumed, etc.)
4. Other
5. Provenance data (e.g. records describing the origins of data or events, the flow of data, how data was collected, any pre-processing, etc.)
6. Product/system data (e.g. data from sensors, peripheral devices such as mouse or keyboard, etc.)
7. Error/crash data (e.g. crash reports, error logs, etc.)
8. Input data (e.g. uploaded files, speech, text input, etc.)

Q12 - Which other types of data are recorded during design/development/testing?

Q13 - What are the reasons for recording data during design/development/testing? Check all that apply.

1. To verify correct operation (e.g. according to a design specification)
2. To improve performance
3. To identify bugs/issues/failures
4. Other
5. To demonstrate compliance (e.g. with regulations such as the General Data Protection Regulation)
6. To track design/development/testing progress
7. For insurance purposes

Q14 - What are the other reasons for recording data during design/development/testing?

The following questions relate to data recorded from products while being used by customers. Is any data recorded while customers are using your products (i.e. during operation)?

Q15 - What types of data are recorded while your products are being used (i.e. are being operated)? Select all that apply.

1. Product usage data (e.g. app start and end time, user preferences and settings, etc.)
2. Error/crash data (e.g. crash reports, error logs, etc.)
3. Meta information (versioning information, operating system, hardware, total resources, etc.)
4. Other
5. Provenance data (e.g. records describing the origins of data or events, the flow of data, how data was collected, any pre-processing, etc.)
6. Product/system data (e.g. data from sensors, peripheral devices such as mouse or keyboard, etc.)
7. Intermediate/generated data (e.g. interaction events, aggregated data, function outputs, etc.)
8. User input data (e.g. uploaded files, user speech, text input, etc.)

Q16 - Which other types of data are recorded while your products are in use or operation?

Q17 - What are the main reasons for recording data during product usage? Check all those that apply.

1. Advertising
2. To demonstrate compliance with regulations (e.g. GDPR, CCPA)
3. To verify correct operation
4. To identify why a product has failed/crashed
5. To improve performance
6. Detecting potential issues/faults
7. Detecting fraud
8. Other
9. For insurance purposes

Q18 - Please broadly describe the other main reasons for recording data during product use.

The following questions relate to tools and/or libraries used for recording data. Note this is at any stage of the product life cycle, including the during design/development/testing and the use of products.

Q19 - Are any 3rd party (i.e. developed/created/owned outside of your organisation) tools and/or libraries used for recording data?"

1. No
2. Yes

Q20 - Are any tools/libraries used to record data at any stage (design/development/testing/use by customers) developed in-house?

1. No
2. Yes

Q21 - Please elaborate on why 3rd-party tools/libraries are not sufficient. Please also indicate in which phases issues are experienced (e.g. design/development/testing/use).

Q22 - Please elaborate on what aspects of 3rd party tools/libraries are particularly useful for recording data, and in which phases (e.g. design/development/testing/use) the features are most helpful.

Q23 - How are tools/libraries developed in-house better suited to recording data? Check all that apply.

1. They have features not available from 3rd party tools/libraries
2. In-house tools/libraries are better suited for data aggregation
3. In-house tools/libraries are more performant
4. In-house tools/libraries allow for more control over data format and representation
5. Other
6. In-house tools/libraries are better at maintaining consistency

Q24 - You've indicated that there are other reasons why tools/libraries developed in-house are better suited to recording data than 3rd-party ones. Please elaborate in the text field below.