Djakhangir Zakhidov
UX Researcher and Product Manager
Case Study - In Situ Research, UX & Usability
Exploring the viability of phone-based VR for workforce management training
The University of Texas at Dallas (UTD), Texas Department of Transportation (TxDOT)
This research study has been published in Virtual Reality.
Introduction
With this project, I helped evaluate the effectiveness and usability of a VR system for workforce training of TxDOT employees. Our team has developed a Virtual Interactive Management (VIM) training simulation which delivers the content from the traditional Management training courses offered at TxDOT through a Mobile VR system. Our team has worked closely with Management training instructors at TxDOT to ensure that our training scenarios accurately represent the types of interactions managers could be having with employees.
Before venturing on a path to provide Management training through VR we wanted to evaluate the usability of an inexpensive phone-based VR configuration consisting of an iPhone13 and a Viotek SPECTRE - a plastic phone holder with rubber fittings and adjustable straps to go around the user's head. We recruited 20 students from the school of management at UT Dallas and measured usability of our system using the System Usability Scale. High SUS scores (above 80) provided evidence and support for the usability of the Management training using even the inexpensive SPECTRE phone-based VR configuration. We felt confident in presenting this affordable solution together with more expensive VR headsets: the META Quest 2 and VIVE Focus 3.
With the current study we wanted to explore the usability of several VR systems and potential relationships between the usability and ergonomics of VR devices, and also look at user receptiveness to VR content. Another key goal was to test these VR systems in their intended use environment - in the classroom. For this work we define "in-situ" to mean a learning activity that takes place in the location or environment where the participants will actually be trained. All of our experiments were in-situ. Our findings indicate that VR is an effective and engaging solution for workforce and management communication training, and that hardware configurations influence overall employee enthusiasm for VR-based training.
SPECTRE/iPhone
META Quest 2
VIVE Focus 3
Exploring three VR Configurations
One of our main goals was to understand the tradeoffs between three potential VR solutions:
SPECTRE/iPhone 13
-
Least expensive - $30
-
Prior usability study established good usability - SUS score of 80 or above.
META QUEST 2
-
Affordable - $400
-
Ease of use, adoption by gaming community
VIVE FOCUS 3
-
The most expensive - $1,200
-
Advanced functionality, such as s biometrics
In-Situ Research & Design-based Research
Our experiment design was influenced by the need to conduct our tests "in-situ" - or in the actual classrooms where TxDOT employees received Management training. We took our prototype to TxDOT offices in Austin, Yoakum, and Dallas, Texas. We successfully ran several multi-day studies, but in those cases we always lost some participants toward the end due to attrition. The chart below represents our experiments in different cities and the changes in participants we experienced.
Results
The detailed experiment procedures and results are described in the paper we just published. Here I would just like to reflect on several interesting findings:
Quantitative Data Analysis
Overall across all three devices, employees reported that the VIM system successfully met each module’s learning objectives. Aggregate means across seven management techniques ranged between 4.12 and 4.84, indicating high satisfaction with VR systems. I then ran a series of one-way ANOVAs to see if there was a statistically significant difference in the ratings of management techniques when employees used a different device: iPhone/SPECTRE, META or VIVE. A series of one-way ANOVAs, equal variances assumed, revealed that for all but one of the seven technique ratings there was no statistically significant difference between the iPhone/SPECTRE, META, and VIVE groups at a 95% confidence level. I interpreted this to mean that:
Across all three devices, users felt more or less the same about the effectiveness of VR to successfully meet each content module's learning objective.
Similarly, we received high aggregate means for UI element ratings across all three devices, and then one-way ANOVAs revealed that there is no statistically significant difference in UI element ratings between the three VR systems.
System Usability Scale (SUS)
What we found surprising was that even with the high ratings for UI elements and the ability to learn from the content presented by the VIM system, the SUS scores across the three devices were quite low. And these low SUS scores were not significantly different between the three devices.
Why were the SUS scores so low, compared even to our earlier study? Could it be because we had younger participants in our earlier study? Or could there be an age group or a gender that was influencing the overall scores? So I decided to analyze the results by demographic factors. However, the demographic analysis of the VIM system’s SUS ratings by participant age, gender, employee role, ethnicity, and education level shows no statistically significant difference at a 95% confidence level within each device group. Employees across all five demographics groups provided consistent device ratings.
Qualitative Data Analysis
We decided to look at qualitative data for an explanation - and we found it!
Consistent qualitative feedback early in the study suggested that the META system’s lack of adjustable head straps had a negative influence on its usability. In general, all three devices suffered various ergonomic issues, yet this issue with META's factory head strap kept coming up in qualitative feedback we collected from TxDOT employees.
So, we replaced the standard head straps on the META with a 3rd party strap which consisted of foam padding, adjustable plastic-based straps, and an over-head elastic band to evenly balance the headset's weight across users' heads and faces to ensure long term comfort and minimal shifting from the device. This change resulted in a significant increase in the SUS score, from "Below Average" to "Excellent":
Overall, in the qualitative comments, employees could give both positive and negative comments. From a holistic positive perspective, many employees viewed VIM as a beneficial, flexible form of individualistic learning. Respondents enjoyed the ease of use and novelty of the technology’s potential to make training more interesting, fun, and exciting. One employee stated: “This is so much better than most classroom type classes. Being new adds some excitement to the learning experience. This type of learning can be beneficial because it eliminates the possibilities of bad teaching styles. All the learning will be the same for each student.” Another employee commented: “It would be a nice idea for all of the people becoming a supervisor to take this for training.” Another employee stated: “This allows you to feel more comfortable without the true human interaction.” A couple of employees also commented: “This would be a good way for individual employees to get necessary training and eliminate travel and space issues.” “I think that this equipment would be useful in training or dealing with life like experiences. It is individually based. I prefer visuals compared to just talking.”
Employees also provided critique on multiple aspects of the VIM system. A few employees mentioned that interacting with the simulation felt awkward and less than ideal. They found it challenging to set up the program without assistance, even if they felt they possessed technical skills. Some employees stated that they experienced feelings of sickness, eye strain, and difficulty maintaining focus. Additionally, comments were made regarding the practical use of the system compared to traditional training methods. One employee also commented on the learning module content, expressing dissatisfaction with the leadership interaction styles module. The comment highlighted issues with the content's writing and the limited usability of the model. The rest of the comprehensive qualitative data analysis is presented in our recently published paper.
So, What Have We Learned?
-
These experiments indicate that user ratings of the virtual curriculum content remained mostly independent from user ratings of the physical device on which the curriculum was presented.
-
Another important finding for TxDOT was that field-and-office-based employees showed no significant difference in their willingness to accept the VIM system as a viable training tool.
-
What I think is the most important revelation of this research is that device usability can be significantly improved through inexpensive modifications to factory hardware. For example, when we replaced the META Quest 2 factory strap with a 3rd party adjustable head strap, SUS score increased significantly from "Below Average" to "Excellent".
-
Following this adjustment to the head strap, the META became the most highly rated configuration among users for implementation of the VIM system, due to its improved usability and comfort. In this configuration we also obtained a statistically significant improvement in participants’ willingness to recommend the system to others with the mean moving from 3.61 to 4.40 following the strap replacement, and the mode score rising from 3 to 5. This work highlights the importance of hardware comfort in VR-based content delivery and makes a case for using iterative, research-based design for VR system development.
-
Comprehensively, these findings suggest that the modified META version of the VIM system would be the best suited model for implementation in a face-to-face, classroom environment. For home-based distribution, the iPhone/SPECTRE version of the VIM system appears optimal based on the ability to get the device to employees in a work from home modality. However, a quality review would likely need to occur to ensure SPECTRE headsets are all consistently functional--especially with the selection button.
Final Thoughts
Unlike many existing studies on VR-based applications that have taken place in isolated, laboratory, or theoretical settings for the sole purpose of investigation, this experiment addresses field implementation and distribution of VR-based training resources as an immediate workforce training tool and considers many real-world factors in situ. I feel that this approach is valuable in that we encountered many of the more subtle issues beyond content that could affect the overall acceptance of the VIM system, most notably some of the hardware reliability and ergonomic issues. While doing this work in situ was challenging in terms of employee availability, attrition due to work schedules and sample size available, we feel confident that this was the optimal way to do this work and uncovered many important issues that will be important in supporting the diffusion of VR into workplace training.