Augemented Reality - An Interview

Laboratory workflows can be time-consuming and tedious. It’s even worse when you are busy, you are wearing gloves and your hands are contaminated. You have to take off your gloves to type on the computer. Then you put them back on. Then you take them back off to type something else on the computer. This continuous switching back and forth disrupts your workflow, making your task even more labor-intensive.

Jamie Kunzelman, Evan Amason, Rachel Sun and project manager Parnika Sharma—all of whom graduated this spring from Harvey Mudd College (HMC) in Claremont, California—decided to explore this challenge and a possible remedy during their 2019 Clinic Program in Computer Science.

The Clinic Programs in Computer Science, Engineering, Mathematics, Physics and Multidisciplinary Studies have been a hallmark of HMC for more than 50 years—engaging juniors and seniors alike solving real world, technical problems in partnership with corporate sponsors. Students work in groups of four or five under the guidance of a faculty advisor and dedicated liaisons from their sponsoring organization. The Clinic Project counts as a 3-credit course involving about 1,200 to 1,500 work hours.

As they contemplated their senior capstone experience prior to graduation this year, the “Sharma Team” was assigned to work with BIOVIA Dassault Systèmes. Their BIOVIA liaisons were Ian Kerman and Dr. Reza Sadeghi who suggested that they focus their project on “Improving Laboratory Outcomes with Augmented Reality.” The team’s faculty advisor, HMC biology professor, Dr. Elliot C. Bush (now Chairman of the Biology Department), agreed that this would be a valuable topic to explore.

Keeping this motivation in mind, we started by developing a more specific problem statement. We set out to improve consistency within the laboratory through a proof-of-concept, augmented reality application that combined a digital, cloud-based lab notebook and a virtual lab procedures.
by Jamie Kunzelman

After interviewing numerous lab science professors at HMC to learn what their major pain points were and researching various augmented reality (AR) technologies, the team decided to move forward with the Microsoft HoloLens AR headset. “We chose the HoloLens because it’s developer-friendly; it uses a popular game engine that we were familiar with, and there’s lots of online documentation,” continued Jamie. “The HoloLens also has an active developer community and built-in speech and gesture recognition capabilities that we use in our application in the form of hand gestures and voice commands.”

Evan Amason took a deeper dive into the AR application they developed. “We split our application into two parts, creating a digital laboratory procedure for weighing a sample along with a digital laboratory notebook that’s integrated with BIOVIA ScienceCloud. We wanted to simplify and streamline note taking without interrupting the workflow. At any point in the procedure, you can quickly access your notebook with the spoken command ‘Open Notebook.’ Then our HoloLens speech-to-text feature allows you to record notes such as, ‘The aspirin sample synthesized was 3.86 grams.’ Then you say ‘submit,’ and the application makes an API call to BIOVIA Pipeline Pilot, which sends the information to ScienceCloud and returns a success message. Our application is touch-free, meaning you don’t have to physically touch anything to navigate the application. This is especially helpful in the lab. If your hands are contaminated, you never have to take off your gloves to flip through your procedure or record your notes.”

Jamie and Evan presented their team’s work, including a brief demo of their app, at a plenary session of BIOVIA’s North American User Conference in San Diego, CA this June. They also participated in the exhibit hall at the User Conference. A large number of interested conference attendees had the opportunity to try on the HoloLens and enjoy the experience of hands-free experiment documentation at the bench.

BIOVIA Dassault Systèmes would like to thank Jamie, Evan, Rachel and Parnika for their hard work and dedication in developing a proof-of-concept laboratory app that is central to the digital lab experience that BIOVIA offers its own customers.

Since graduating from HMC in May, Jamie has moved into technical program management at Microsoft. Evan is working for a healthcare software company where he looks forward to developing customer-facing software and “creating good experiences for non-technical people.” Jamie and Evan both enjoyed their AR project with BIOVIA. Jamie’s favorite part was being creative and innovative—figuring out the problem space, narrowing it down, ideating and building a viable solution. Evan enjoyed the cutting-edge nature of the technology, and he would be interested in working with AR in the future.

Both agree that their clinic program and collaboration with BIOVIA have been valuable experiences as they move from college to today’s challenging industry environment.