The University of Texas at Dallas (UTD), The University of Texas Southwestern Medical School (UTSW), Arizona State University (ASU). Four-Year long project, sponsored by the National Science Foundation (NSF).

 

The full research study has been published in the International Journal of Human-Computer Interaction in March of 2024.

​

Introduction

​

I am proud to have been appointed the project director on this federally-funded simulation R&D project as it gave me an opportunity to apply my expertise in user research, user-centered design, iterative development, evaluation, and management during the development of this one-of-a-kind simulation product.

​

From the onset, we were faced with a complex technical challenge - the development of a conversational virtual patient in augmented reality. And by the end our system has exponentially grown in complexity to also include additional virtual humans and additional interaction modalities - co-located AR and distributed AR.  I am so fortunate that I got to lead a team of extremely skilled and talented interaction designers, software developers, UX researchers, and cognitive scientists during the development of our prototype.  I attribute the success of our prototype to the joint efforts of our team at the Center for Simulation and Synthetic Humans at the University of Texas at Dallas, our colleagues and subject matter experts in the Office of Medical Education at the University of Southwestern Medical School, and learning science experts at the Arizona State University. 

 

How it All Started - Understanding the Challenges in Medical Education

 

Our research lab prides itself on being the center for excellence in simulation research.  In 2016 we have developed and tested an online learning portal for pre-med students.  During this time we have developed a relationship with the dean of undergraduate education at the University of Texas Southwestern Medical Center, Dr. Robert Rege. Dr. Rege was impressed with our early efforts to improve the communication skills of pre-med students and educate them about the dangers of unprofessional behaviours on social media.  I presented the results of this early research at the 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH), which was published in the conference proceedings.

​

This was the time when we started discussing more advanced solutions to the challenges faced by young medical school students. One of the key challenges that the students faced was a lack of opportunities to practice medical interviewing. It turned out that the medical school trained and hired special actors, known as standardized patients, to portray sick patients.  This presented a variety of challenges, including the logistical challenges of scheduling patient interview sessions, the need for students to often practice in a group, and the inconsistency in the quality of the standardized patients.  As a result, students would only get two or three patient interview sessions during their time in the medical Pre-Clerkship, which includes the 1st year and 1st semester of the 2nd year in medical school. 

​

Armed with this knowledge I conducted additional interviews and focus groups to better understand our users and their needs.

​

Strategic Thinking

​

Product managers need to look beyond short-term goals and envision where they want the product to be in the future. To accomplish this, I started by understanding market trends, technological advancements, and customer needs to formulate a long-term vision for the product.

​

With guidance from Dr. Marjorie Zielke, the director of our research center, I reviewed the market for simulation products and identified a huge gap in the space of medical communication training. While a number of conventional products existed in this space, none offered what we proposed to develop - a human-like conversational virtual patient trainer anchored in the confines of an actual patient room thanks to the unique affordances of spatial computing and augmented reality.

Technical Aptitude

It was important for me to possess a fundamental understanding of all the technologies involved in the development of the product that we devised. I already had several years of experience developing interactive simulations using game engines in virtual reality, but I had to spend some time learning the new affordances of augmented reality.  What was even harder for me was understanding the basics of machine learning, crucial for developing the natural language processing algorithm for imbuing our virtual character with conversational capabilities.  Well, a good way to learn something is through teaching. I had the opportunity to produce and present several research posters on the topics of augmented reality, natural language processing, and animated virtual characters at the Smart and Connected Communities Conferences in Austin, TX, and Denver, Co.

User Needs

I wrote a separate case study on all of my UX research efforts on this project, including identifying who our users are, understanding their needs and using user-centered design principles to design an optimal learning experience.  In summary, our users were 1st and 2nd grade medical students needing a better way to practice medical interviewing.  This was a highly competitive and motivated group who saw the value in our product and was willing to take the time to explore a new learning tool.  Our users' key needs were: 

• Students need a low-stakes environment for practice

• Students need individual AND group practice opportunities

• Students need demonstrations of proper technique and feedback

• Students need to be able to practice as much as they want, wherever and whenever

UX Design

Our UX Design was driven by user needs. As such, we designed a system in which the students could:

1. Interact with a novel AR user interface using voice and gestures

2. Learn through observing a medical interview demonstration by a PAL or a human collaborator

3. Learn through interviewing the VP by speaking naturally and asking questions just like they do with a standardized patient (SP)

4. Learn through receiving feedback from a PAL or a human colleague.

In addition, I helped conceptualize and develop two novel interaction modalities for social learning — Co-Located AR and Distributed AR, so that the students could:

1. Learn on their own or together with a co-located human colleague.

2. Receive feedback from a remotely located human expert.

Goal Setting, Product Roadmap & Prioritization

I realized fairly quickly that to accomplish our feat we would need significant funding and support beyond the resources of our research lab.  This prompted me to hone my skill in grant-writing, and I am proud to say that after several failed attempts, I helped to co-write a proposal that was accepted and granted by the National Science Foundation. In addition to having Dr. Rege on our team, Dr. Zielke (my boss) succeeded in bringing another important team member on board, Dr. Scottie Craig, an expert in learning science.  We now had a stellar team and funding, we just needed to actually build the darn thing. 

​

As I got into developing our product roadmap through detailed gantt charts and lengthy discussions with our engineers and researchers I quickly realized that I will need to learn how to prioritize tasks due to limited time and resources.  After all, we were only a university lab at best resembling a small-size startup.  I ended up producing a product roadmap that enabled us to quickly assess the feasibility of several potential approaches and eliminate the wrong ones, focusing our energy on the correct path, thus saving us significant time and resources.  My roadmap enabled iterative development along several parallel tasks: natural language processing, base networking architecture, co-located and distributed AR modalities, unity scene development with custom C# code, and the full art production pipeline. 

​

Iterative Development

​

Without realizing what I was doing I started utilizing Agile and Scrum software development methodologies which enabled me to effectively lead the iterative development process through sprint planning, sprint reviews, and various testing sessions.  It was crucial to be receiving continuous feedback from all the stakeholders involved in this project because as I learned quickly, engineers and designers don't think alike. We had to make a lot of decisions quickly, and it was paramount to communicate important choices throughout the whole team. 

​

My time spent in researching and learning the technical stack of our product paid off - I started getting a handle on the programming languages we used, various ML frameworks, training and testing datasets, databases, networking and infrastructure components.  This knowledge helped me grasp the technical complexities involved in building, testing, and evaluating our product.

​

My background in video production and motion capture helped me ideate and lead the production pipeline of animations needed for our virtual characters. I organized and led the filming sessions where we captured human actors deliver various lines representing the responses from our virtual characters.  We then edited the footage and fed it into Faceware - a facial motion capture software that applied markers to the faces of our actors and then extracted facial movements and applied them onto 3D characters that we developed in Maya.  

Risk Mitigation and Problem Solving

Our development suddenly came to a halt when the COVID-19 pandemic hit in March of 2020. Nobody anticipated this disruption and at first it took us by surprise.  We had to pack up our computers and switch to a work-from-home environment. This meant no more face-to-face meetings and more importantly, no in-person access to medical students. This was especially troublesome since we were just starting to build a dataset of doctor-patient conversations to train our NLP system. We had to come up with a solution fast.  This is when I trusted the intuition of our developer and supported his suggestion to build a web-based chatbot version of our virtual patient.  Just as with everything, his initial estimations of how long this would take were wrong and it took about twice the amount of time that I anticipated.  However, it was the right decision after all as it simplified data collection and sped up the creation of the dataset that we needed.  

​

Another problem was conducting human-subjects experiments at a medical school which is also a hospital during a pandemic. We had to be extra careful not to get sick while conducting experiments in a simulation center located on a 2nd floor of a large Dallas hospital. What I also didn't anticipate is just how long it takes to get human experiments approved by an institutional review board (IRB) during a pandemic. But nevertheless, we persevered and worked it all out in the end, learning in the process that Microsoft HoloLens 2 built-in microphones are able to pick up a user's voice just fine even when he or she is wearing a face mask.  However, the microphones struggle to get clean audio with folks that have long or abundant hair because the hair tends to cover up the mics, which are located on the side walls of the AR device.  

Leadership

Being a good leader is hard because it involves a lot of flexibility and adaptability on the part of the product manager.  There were times when I could fully trust my team to make certain decisions, empowering them to take ownership of their work.  Yet there were also times when I had to question what they were working on and nudge them in a different direction.  It was important to establish transparency and frequency in communications between all stakeholders to create a kind of environment where everyone understood the product vision, goals, priorities, responsibilities, and ongoing progress. 

​

I understood the need for cross-training of our team members to build bench strength for our lab. I took it upon myself to spend time to provide guidance, mentorship, and coaching to help our team members develop new skills, grow professionally, and grow outside of their comfort zone.  This has paid off on so many levels because by the end of product development our artists and engineers could be moved onto research-related tasks such as data collection, analysis, and scientific writing. Practically everyone on our team had to learn how to conduct a literature review, help write grant proposals, and review and edit other people's work.

 

Finally, I believe good leaders are great leading by example. I took pride in mastering a variety of skills and wearing lots of different hats on this project. For example, I took two graduate courses in statistics at UTD to both fill some knowledge gaps and set an example for other team members to follow.  As a result, several other team members continued their education through taking additional courses.

Impact

I provide a detailed breakdown of the impact of this early-prototype product on the users - both on the overall user experience and its' effect on confidence and knowledge of medical interviewing - in a separate UX Case Study. Also, we recently published the full study results in the International Journal of Human-Computer Interaction.

​

These results are very promising and provide strong evidence in support of novel learning simulation tools that we will see emerge in the near future.  Social interactions with AI-driven conversational and human-like virtual humans in augmented reality won't just be limited to medical education but will undoubtedly affect all spheres of education, including STEM, continuous learning in formal and informal settings, and entertainment.  Our work provides a one-of-its-kind framework for trying out novel configurations of human-machine teaming: one-on-one interactions with visualized AI, interactions with a co-located human peer or remote experts, and most importantly, an environment for testing when a virtual human is needed vs. when a real human is needed. 

​

This is just the beginning and we are looking for ways to take this early prototype further and develop market-ready products. Please contact me to discuss how we can collaborate.