Augmented and Virtual Reality Apps for Training

Augmented reality (AR) and virtual reality (VR) find important applications in professional training, particularly for doctors. The value and complexity of medical training justify the cost of the head-borne devices. This article explores research at Inteleos to improve the use of augmented and virtual reality, which I’ll talk about together under the commonly used abbreviation XR. We’ll also look at XR applications in health care by Mobiquity.

How XR is Trained

We all learn from other people, even if their expertise is filtered through books, computer applications, or even generative AI. So medical training starts with the medical expert.

At Inteleos, I interviewed Heather Harris, Ph.D., a data science consultant with Herkimer Consulting, and Inteleos data scientist Denali Carpenter. The two scientists are researching the use of process mining to develop a framework for simulating healthcare assessment tasks in VR.

Process mining has traditionally been used in business to capture workflows and processes for analysis (e.g., task completion, steps for completion, timing to completion, etc.). By capturing this data, businesses can make informed decisions to optimize the timing of tasks and other aspects of business.

Process mining can be used for medical imaging education and assessment after collecting data on “expert” users. For example, sonography experts who have already passed their certification exams might complete simulated VR tasks. The similarity in performance patterns can then be used to create profiles of “optimal” performance, or behaviors that are expected to be performed in order to successfully complete a task such as a sonogram of the heart.

Once a profile is created, on the simplest level, a VR headset, controller, and sensors track student activity and match the outputs to the desired actions performed by the expert. If a student puts an instrument in a suboptimal position, for instance, a notification displayed in the headset will recommend an adjustment for positioning the probe.

Harris and Carpenter explained that their process mining approach would record two types of data to create optimal profiles. “VR allows us to collect two main streams of data: the motions of the expert doing the work and the output—in this case, a medical image,” said Harris. “we found that a 20-minute VR session With just two streams produces, on average, two million data points from a user. Formative feedback, based on expert judgment via process mining, can then allow developers in medical imaging education and assessment to build tasks using these processes and improve overall mastery.”

“The main goal of our research is for people to understand the current state of VR, the limitations, and the role of data science,” said Carpenter. “We noticed a gap between the VR systems and the actual data science. Data is comimg in constantly, but few people are taking that data to develop software that can then inform the system. Our goal is to provide sound recommendations for data science to grow within the VR space.”

XR for Disaster Training

Mobiquity has developed XR apps to train emergency responders for disaster situations. The ability of an AR app to impose realistic artifacts on the student’s environment, and the ability of VR to create an emotionally effective simulated environment, make these tools valuable for training. According to Teun Schutte, strategy director of life science and healthcare at Mobiquity, AR can be better than VR, depending on the training goal, because the student can interact with real-world objects and other people. But VR creates a more immersive experience.

Both visual and auditory stimuli can prepare a student for a fire, flood, etc. Schutte says that students undertaking XR training report that they feel the situation is real.

Other uses for XR include:

  • Putting a student in the role of a patient to sense what’s like to lie in bed and be treated.
  • Training students to talk to different kinds of simulated patients, to help them improve empathy and their interview methods.
  • Training patients in motor or sensory skills lost due to accident or illness, to help them with rehabilitation.

Devices used by Mobiquity include Oculus Pro, HTC Vive, smart glasses, haptic gloves, and handheld devices. Schutte says that apps usually require 5 to 10 megabytes per second of network bandwidth.

Simulations are never as accurate as engaging in an accurate medical practice. But neither are more traditional forms of education. Thus, XR should help to train more people more quickly to be ready for their first real-life service.

About the author

Andy Oram

Andy is a writer and editor in the computer field. His editorial projects have ranged from a legal guide covering intellectual property to a graphic novel about teenage hackers. A correspondent for Healthcare IT Today, Andy also writes often on policy issues related to the Internet and on trends affecting technical innovation and its effects on society. Print publications where his work has appeared include The Economist, Communications of the ACM, Copyright World, the Journal of Information Technology & Politics, Vanguardia Dossier, and Internet Law and Business. Conferences where he has presented talks include O'Reilly's Open Source Convention, FISL (Brazil), FOSDEM (Brussels), DebConf, and LibrePlanet. Andy participates in the Association for Computing Machinery's policy organization, named USTPC, and is on the editorial board of the Linux Professional Institute.

   

Categories