Skip to main content
The Washington PostDemocracy Dies in Darkness

From surgery simulators to medical mishaps in space, video-game tech is helping doctors at work

Training platform Osso VR is one of many emerging medical technologies to come from virtual reality. (Osso VR)

Justin Barad never would have guessed that he would help operate on a gorilla during his residency at the UCLA Medical Center, where he was training to be an orthopedic surgeon. But in August 2014, the Los Angeles Zoo and Botanical Gardens needed specialists to help Jabari, a 400-pound gorilla who was limping and unable to put weight on his leg. The surgery was successful — but it was a challenging endeavor for a team that didn’t normally operate on animals.

The zoo’s request was an unusual one, even by industry standards, but unpredictability is commonplace in the medical world. His experience with Jabari is one of many reasons Barad sought a solution for surgeons and doctors so often faced with unknowns. These moments drove him to dedicate his career to making surgery training more efficient by using video-game technology.

Along with a team of software developers and medical experts, he founded Osso VR in 2016. It’s an attempt to curb unpredictability, giving surgeons on-demand information via a virtual reality headset and tools to properly assess how to best handle surgery. Its success is hard to ignore: It was one of Time’s best inventions of 2019, and according to a study by researchers at the University of California at Los Angeles, training with Osso VR is 230 percent more efficient than traditional methods. Despite launching just three years ago for the Oculus Rift, the virtual reality-based surgery training program is now used to train over a thousand surgeons a month worldwide.

Other training methods range in cost and effectiveness. Some surgeons use simulation centers, which are big, bulky machines used to simulate surgeries, but access to them is limited. Many residency programs and medical schools don’t have one, because they are extremely expensive and require adequate space; and even at schools that do, certified doctors can’t pop back into a simulation as easily as medical students can. Moreover, because the machines tend to be old, simulations are far from modern. Students practice on cadavers and animal labs more than they use virtual simulations.

But even with these resources, surgeons still make frequent mistakes: A 2015 study conducted at Massachusetts General Hospital showed that its staff made drug-related errors in 50 percent of its surgeries.

Osso VR is an attempt to bridge the gap, elevating the quality by bringing antiquated simulations to the modern era.

“If there was a way, kind of like in the Matrix where you just plug in for five or 10 minutes and refresh yourself or get up to speed and then assess and make sure you’re ready to go, that would be a total game changer and have a massive impact on patients and the health care system as a whole, all thanks to video game technology,” Barad told The Washington Post in a phone interview.

The 10 best VR games of 2019

Osso VR may not exactly be equivalent to plugging in to the Matrix, but it allows surgeons to hone their skills by operating on virtual patients with tools they would normally find in an operating room. This lets trainees experience it first-hand without the fear of error. Right now, Osso VR is mainly used for orthopedic procedures, but Barad’s team is working to expand into other areas, such as vascular, heart, urology and surgeries conducted with robotics.

Barad says surgical training is still “primitive,” relying heavily on learning through observation. He says it was during his residency at UCLA where he “experienced firsthand the biggest problem facing medicine today.”

“I’d be in on a surgery and they’d be like, ‘Hey, get up and go run to the computer real quick and Google what we’re supposed to be doing,’” he said.

Medical students would look up YouTube videos — the platform has become a popular resource for surgeons in training, with tens of thousands of videos on the subject — for guidance to prep for an operation. Sometimes Barad and his colleagues would even watch one in the operating theater if the surgery was especially complicated, or sprawl out an instruction manual to better understand medical devices.

Osso VR has a multiplayer component, too, that allows surgeons to work together in virtual reality. Barad compares surgery to a “team sport.”

“It’s like soccer or like a symphony,” he said. “With VR, you can get everyone in the same virtual operating room so you can train as a team and you can practice knowledge of technical skills.”

Because of surgery’s unpredictable nature, and because the bulk of its training relies on hands-on experience, Barad said, the only time he underwent a “technical assessment” was when he was asked to play the board game Operation during his residency interview. Part of the problem, Barad said, is that there are no “scalable, low-cost ways” to easily measure surgical skill. Osso VR offers a solution: It has built-in analytics that objectively measure surgical performance.

Dept. of Veterans Affairs believes games can help soldiers reconnect, reduce suicides

Osso VR is one of many emerging medical technologies to come from virtual reality. “Sea Hero Quest,” for example, is a VR game that has been used to research the early signs of dementia, and “SnowWorld” was one of the first virtual reality simulations to help burn victims with pain management; studies say it’s more effective than morphine. But virtual reality is far from the only player in the future of health-care tech.

Saving lives by playing mobile games

Schell Games, a studio based in Pittsburgh that built the VR action puzzler “I Expect You to Die,” has regularly produced games for education and entertainment. The bulk of its library is for a younger audience, and when it has tapped into the medical field, it has been mostly for patient-facing technology, such as therapy for children on the autism spectrum. With its new project, “Night Shift,” Schell Games pivoted toward making a mobile game designed specifically for emergency room physicians.

Schell Games is working closely with Deepika Mohan, a medical professor at the University of Pittsburgh, to develop “Night Shift,” a training simulator designed as a story-driven adventure game that puts players in the shoes of a trauma doctor. It is currently in closed beta and accessible to a limited number of physicians before its launch.

The response from doctors has already been positive. In 2017, Mohan published her research and shared that doctors who played “Night Shift” outperformed those who did not.

“One of the big decisions that doctors have to make when a trauma comes in is, do they deal with it there or do they send this person to trauma center?” said Jesse Schell, the chief executive of Schell Games. Mohan’s “observation was that doctors don’t get very good feedback about whether they made the right decision or not.”

The idea was born out of Mohan’s struggle to make training engaging, when past efforts had failed. She had a hypothesis: Would an interactive story with emotional beats to accompany training make it more memorable and effective? Night Shift is proof of that concept. Its physician protagonist moves to a new town and works in the emergency room. You spend shifts diagnosing and treating patients, then deciding whether their care is critical enough to transport them to a trauma center. But there’s more to this world than just the hospital.

It’s not just any town,” Schell said. “It turns out it is the town where [the protagonist’s] grandfather had lived. And part of the reason he’s moving there is because his grandfather had recently passed away and he’s moving there to help settle his grandfather’s affairs.

Schell calls Night Shift an “interactive soap opera” with elements of mystery and adventure. Trying to understand who your grandfather was plays a big part, but you also develop relationships with your patients.

“You get to know some of the patients," he said. "If you end up making misdiagnoses and put them in a bad way, you see them suffer for it or sometimes you see them pass away, and then you’re confronted with their sobbing relatives.”

While Night Shift has more of a cartoon-y aesthetic, Level Ex is a Chicago-based studio making mobile games for doctors, which feature “hyper-realistic” simulations that physicians can play on the go.

Level Ex CEO Sam Glassenberg has been in the video game industry for two decades. Despite coming from a long line of doctors, he decided against going to med school, making him the “black sheep” in his family. Instead, he became a video game developer.

Glassenberg worked at LucasArts making games for the PlayStation 2 and the original Xbox, had a stint at Microsoft where he led the DirectX graphics team, and worked with publisher FTX Games to produce mobile games based off popular Hollywood movies like the Hunger Games and Mission Impossible. But things changed when his father, an anesthesiologist, requested that he make a game to help train his colleagues in fiberoptic intubation (a technique that opens up a patient’s airways). With his father’s help Glassenberg “threw together” the game, called iLarynx, in a matter of weeks and published it to the App Store. To his surprise, it was so popular that a hundred thousand medical professionals were still playing it two years later.

Clearly there was demand for this," he said. "[I wondered,] what if instead of just throwing something together, you put a team of top video game developers, designers and artists on the problem of capturing the challenges of the practice of medicine in video games? That’s what started Level Ex.”

Now, half a million medical professionals worldwide are playing Glassenberg’s free mobile games. Level Ex has a team consisting of video game industry veterans, medical experts and software engineers. The studio has created four games, each with their own focus on a different specialty: pulmonology, gastroenterology, anesthesiology, and cardiology.

”Studies have shown that if a surgeon plays an action game before they go into surgery, they will perform better at laparoscopic and endoscopic procedures, even if that game has nothing to do with their actual procedure. So we know that there is a lot that can be done there," Glassenberg said.

Level Ex’s realistic simulations — which run impressively well on a smartphone — let you see inside the human anatomy as you play through different cases, which involve scenarios like removing a foreign object from a patient’s airways or performing a colonoscopy on a virtual Crohn’s Disease sufferer. The better you are, the more you progress. And as you progress, you unlock new levels, tools and techniques. It was also important to Glassenberg to bring high graphical fidelity to his games, since most antiquated surgical simulators “look like a video game out of 1995.”

Launching games into space

Video game tech can also assist with highly irregular situations or those that would be impossible to replicate. For instance: What happens when an astronaut gets injured in space? Flight surgeons help in training and advising, but they’re stuck at the NASA Mission Control Center rather than on the spacecraft. For astronauts, the onus is on them to be informed and take the right precautions and steps to stay safe.

For example, if an astronaut clutches his chest and falls unconscious in zero gravity, then he needs to be brought over to an ultrasound as a radiologist in Houston navigates the situation.

“An ultrasound is the only visualization you can do [in space],” Glassenberg said. "But the heart physically changes shape in micro-gravity depending on how long you’ve been up there. So how do you know what you’re looking at is normal for somebody who’s been in space for nine months or if it’s abnormal?”

Space travel is challenging enough, and medical emergencies make it even tougher to navigate. That’s why NASA asked Level Ex to build a video game simulation that shows “the human body’s physiological and anatomical changes” in space, and what procedures and devices are necessary to address them. This project is still underway and not yet available to NASA, but with a significant grant from TRISH (the Translational Research Institute for Space Health), Glassenberg is positive his team is equipped to make it happen.

Glassenberg says he still has “a million questions,” but he’s working hand-in-hand with TRISH, NASA’s medical teams and astronauts to figure it out as best he can.

“We’re just one part of that puzzle,” he said.

Looking to the future

As Level Ex focuses its ambitions on outer space, Osso VR is keeping its goals firmly within virtual reality. But the team still has big plans for 2020 and beyond.

Osso VR isn’t yet available on Oculus Quest, which is a powerful and wireless headset untethered from consoles and PCs, but it will be coming to the platform this year. When it does, surgeons will be able to bring a portable (and affordable, in comparison to the aforementioned traditional simulation centers) VR headset to any operating room.

Barad is intrigued by Oculus Quest’s hand-tracking innovations, which rolled out late last year, since surgery requires fine motor skills and accuracy to do right.

“There’s a lot of interest in like gloves and hand tracking and things like that, and those are very exciting areas that are still very early," Barad said. "Using controllers works great for now, but we’re trying to figure out how we can more intuitively get people’s hands and fingers in the mix.”

Like many of his game development peers, Barad grew up on video games, and was coding them since middle school. He even has a game credit to his name (1998′s Heretic II) with Activision. It was his “dream” to become a video game developer, and he also wanted to make a positive impact. That’s where Osso VR came in.

“I’m really grateful to the video game industry, video games and the art form in general," Barad said. “This wouldn’t have been possible if it weren’t for video games and everybody involved in the community that basically got VR up off the ground and into really a mature product.”

Read more:

The hidden world and overlooked problems of acting in video games

Twitter’s most popular game is a Japanese mobile RPG that keeps beating Fortnite

The most anticipated video games of 2020