Using extended reality to transform patient care
On a crisp fall morning, Joaquin Sanchez-Sotelo, M.D., Ph.D., is preparing for shoulder surgery. But on this Thursday, Dr. Sanchez-Sotelo, the division chair of Shoulder and Elbow Surgery at Mayo Clinic, is not putting on sterile garb or scrubbing into the operating room. Instead, he’s seated at his desk, studying a 3D model of the patient’s shoulder on his computer screen.
With a few clicks, he places a digital implant onto the model. With another click, the software runs an artificial intelligence (AI) algorithm that pulls data from the patient’s medical scans and combines it with information about the standard shoulder replacement hardware to generate new specifications for a patient-matched implant. In 4 to 6 weeks’ time, a vendor will deliver a custom 3D-printed device designed to integrate seamlessly with the patient’s body. The goal, Dr. Sanchez-Sotelo says, is to make sure it’s perfect.
In the operating room, Dr. Sanchez-Sotelo dons a high-tech headset. During surgery, the headset projects holograms of the 3D patient model and the custom implant. With a flick of his fingers, he can adjust the image to present new views. The hologram highlights the precise anatomical landmarks detailing where Dr. Sanchez-Sotelo will position and secure the implant.
“It’s like having Superman vision,” he says. “You can see through the body’s structures and see exactly what you’ll need to do during the surgery. This level of detail allows me to complete the surgery faster than ever, and I can place the implant within millimeters and degrees of complete accuracy.”
Technologies like these — augmented, virtual and mixed realities — are revolutionizing medicine across every field from surgery to medical education. As Mayo Clinic works to transform healthcare, scenes that seem like science fiction are already unfolding at all three campuses.
Giselle Coelho, M.D., Ph.D., a Mayo Clinic STR-X (simulation, telemedicine, robotics and experimental education) fellow and pediatric neurosurgeon, has leveraged this technology to address serious brain and skull malformations in infants and young children. In these cases, precision is paramount.
“Pediatric neurosurgery presents unique challenges,” explains Dr. Coelho. “These cases are relatively uncommon, and you may be faced with severe anatomical malformations. You never want the first time you do a procedure to be on an actual child.”
This is where extended reality tools become invaluable. Dr. Coelho’s team combines advanced imaging and 3D modeling technology to transform patient scans into detailed anatomical models.
These aren’t simple visual representations — they’re sophisticated surgical planning tools that clarify important details like the exact location of arteries, the precise dimensions of skull defects, and the spatial relationships between tumors and critical brain structures.
“We can simulate the full surgery before we ever enter the operating room,” Dr. Coelho explains. “The technology allows us to practice our approach, measure exactly how much bone we’ll need for reconstruction, and coordinate with other specialists like plastic surgeons.”
Even more importantly, she says that using augmented reality (AR) headsets projecting holographic images onto the physical model, they can conduct full simulations of the surgery, including nurses and operating room technicians, to ensure that every person knows exactly what to expect before the patient even enters the room.
The impact of this approach is clear. Dr. Coelho shares an example of a complicated procedure her team conducted on a child from the Amazon region of Brazil with an encephalocele — a rare birth defect causing brain tissue to protrude through an opening in the skull. The team used AR technology to plan and demonstrate the surgical approach to the team and used a patient-specific hybrid model to plan and practice the surgery in advance.
“Traditional surgical planning might show us the general anatomy, but with these tools, we could determine the exact measurements of the defect and precisely calculate how much bone we needed to close it,” says Dr. Coelho.
For children requiring complex neurosurgery, this reduction in operating time isn’t just about efficiency. It means less time under anesthesia and potentially better outcomes. Experts say that this technology can transform not just the surgical experience, but the entire trajectory of a young patient’s life.
In the case of her patient with encephalocele, Dr. Coelho says that the procedure completely changed the child’s life, allowing her to excel in her studies and finally be able to fully engage in her community. “That’s what this is all about,” Dr. Coelho says. “We’re using technology to transform possibilities for our patients.”
The term “extended reality” encompasses three emerging technologies: virtual reality, augmented reality and mixed reality.
Illustrations by Jason Schneider
Virtual reality (VR) technologies are fully digital simulated environments, typically accessed via an immersive headset and sometimes including haptic feedback gloves to allow for environmental interaction.
Augmented reality (AR) refers to cases where digital elements are projected onto the real environment, typically using a screen interface or headset.
Mixed reality (MR) combines virtual and real elements, such as in training simulations that project virtual models over real surgical dummies to enhance the learning experience.
Neurosurgeon Rabih Tawk, M.D., believes that these technologies can, and should, be applied to every field. “Every surgeon is always learning,” he says. “Every patient is different, with unique needs. These simulations provide a valuable new platform for surgical planning, tailoring the approach for everyone. Why shouldn’t all surgeons plan this way?”
Dr. Tawk uses a system called Immersive Touch, consisting of virtual reality goggles and hand controllers, to plan complex aneurysm surgeries. These tools allow him to create detailed 3D visualizations of the surgery site using patient brain scans, providing clearer insights into the patient’s anatomy than could be gleaned from simply looking at static 2D angiograms. These insights, in turn, allow him to generate a precise treatment plan.
Even outside of the operating room, these tools are changing how clinicians collaborate to provide care. Neurointensivist William D. Freeman, M.D., listed a multitude of ways that virtual reality is used in his field, including the use of virtual reality to enable rapid case review and consultations with remote specialists — critical for time-sensitive stroke care.
“In some places, hospitals are even using minimally invasive robots and remote controls in collaboration with on-site care providers so surgeons can place stents in stroke patients completely remotely,” he says. “These tools provide opportunities for expert care even in locations where a clinician of the right specialty is not available on-site.”
With the advancement of 6G technology, which allows for nearly instantaneous transmission of high volumes of data, Dr. Freeman sees a rapid evolution in remote surgical collaboration. In the future, he imagines that Mayo Clinic experts will participate in “4D surgeries,” with multiple surgeons in more than one location working to care for a patient together in three-dimensional virtual space, across the fourth dimension of time.
Advancements in data transmission technology will also enable “holoportation,” allowing remote specialists to participate virtually in consultations or medical procedures via realistic 3D hologram projections. At Mayo Clinic, this could mean that a specialist in Rochester appears to be standing beside a surgical team member in Arizona or Florida, providing guidance in real time while maintaining the feel of a natural face-to-face interaction.
While this may sound like science fiction, to Dr. Freeman, it’s simply the next step in ensuring that patients get the care they need, whenever and wherever they need it.
Combined with other advanced technologies, such as AI, extended reality tools are propelling surgery into the next century. “These technologies complement each other perfectly,” explains Dr. Sanchez-Sotelo. “AI helps us see and understand patterns in thousands of surgeries, while extended reality lets us apply those insights with unprecedented precision.”
In practice, this partnership begins long before the first incision. AI algorithms analyze vast databases of surgical outcomes to help design patient-specific implants and suggest optimal surgical approaches based on each patient’s unique anatomy. The technology can identify subtle variations that might affect surgical success.
In the operating room, AI technology processes and interprets imaging data from multiple sources, generating the real-time visualizations that surgeons use via their AR headsets. AI systems can also track and analyze surgical performance metrics, providing instantaneous feedback during training simulations, and allowing trainees and their teachers to track progress against established benchmarks.
These advances are particularly crucial for remote surgery, where Mayo Clinic is pioneering new approaches. Advanced AI filters act as digital stabilizers for robotic surgical systems, eliminating tiny tremors or delays that could impact precision. “These safety systems are essential,” explains Dr. Freeman. “They ensure that even if there’s a minor connectivity hiccup, the robotic system remains steady and precise.”
Like any emerging technology, there are practical challenges to consider — some of which can be addressed simply by changing the room’s lighting. “The system requires specific environmental conditions to work optimally,” says Dr. Sanchez-Sotelo. “Very bright operating room lights can make it difficult to see the holograms, and certain colors of surgical gloves can interfere with the system’s tracking capabilities. You need a reliable internet connection, and you have to be ready to adapt if the technology experiences issues.”
Some more traditional surgeons have expressed concerns about overreliance on technology. Dr. Sanchez-Sotelo emphasizes the importance of maintaining clinical judgment alongside technological proficiency. “You still have to have common sense and be ready to adjust if the plan doesn’t work out,” he says. “These tools enhance our capabilities, but they don’t replace the need for surgical expertise and decision-making.”
In addition, the financial barriers to adoption can be significant. For institutions in underserved areas, these costs can be prohibitive. However, Dr. Freeman points to reduced recovery times as a benefit that offsets these costs. “Preoperative planning reduces complications during surgery and shortens hospital stays,” he says. “Shortening a patient’s length of stay in the ICU by even a day saves thousands of dollars. The cost-effectiveness ratio is quite favorable. These tools are investments that pay off in better patient outcomes and lower overall costs.”
These technologies represent more than just tools — they’re part of a new frontier in personalized medicine. Through Mayo Clinic Platform, healthcare’s first true platform, solution developers are exploring vast datasets and sophisticated AI models to detect diseases earlier, optimize therapies and generate more accurate diagnoses. When combined with extended reality tools, this creates unprecedented opportunities to tailor care to each patient’s unique needs.
But perhaps most transformative is how these technologies are breaking down barriers to accessing expert care. Dr. Sanchez-Sotelo has seen this firsthand. “In many parts of the world, patients don’t have access to surgeons with expertise in complex procedures,” he says. “Now, I can virtually step into an operating room anywhere in the world. I can guide another surgeon in real time, showing them exactly where to place instruments, helping them develop new skills. We’re not just treating today’s patient — we’re helping doctors provide better care for all their future patients.”
These technologies demonstrate that distance need not be a barrier to receiving the highest level of care, that expertise can be shared instantly across continents, and that the future of medicine is both highly technical and deeply human.
“If I’m the patient, I want that top specialist for my critical condition available immediately,” says Dr. Freeman. “These technologies make that possible. They allow us to bring Mayo’s expertise directly to patients, wherever they are, whenever they need it. That’s not just innovation — it’s transformation.”
The Gerstner Scholars Program in AI Translation at Mayo Clinic is accelerating breakthroughs in artificial intelligence (AI)-powered medical translation. Through this program, junior and early-career[...]
AI breakthroughs are meeting patients’ needs sooner.
Kari Turkowski, Ph.D., has never been one to shy away from a challenge.