Contents
On February 9-10, Microsoft held an online broadcast of 12 operations performed in mixed reality. Surgeons from 13 countries have used Microsoft HoloLens 2 hybrid reality glasses. Understanding how it works
How did the operations go?
All surgeries were orthopedic and lasted 24 hours in total. Each of them involved a panel of doctors who used hybrid reality glasses and the Microsoft Dynamics 365 Remote Assist app. This allowed them:
- see all the necessary information – for example, a CT or ultrasound image – in augmented reality (AR);
- broadcast the operation itself to colleagues online;
- share experience with colleagues: get advice and talk about your actions.
In addition to the operation itself, there were 25 online discussions where doctors discussed the latest technologies in medicine. In addition to doctors, more than 70 experts in the field of medicine and healthcare joined the event. The broadcast was watched by 15 viewers from 130 countries.
The surgeons were managed by Thomas Gregory, Professor of Orthopedic Surgery at the Sorbonne-Paris-North University and Head of the Department of Orthopedic and Trauma Surgery at the Avicenna Hospital in Paris. He also performed the first such operation in 2017 – for an 80-year-old patient with a tendon rupture. The operation lasted 90 minutes. In the process, Gregory shared what was happening with four other surgeons in the US and UK via Skype.
Hololens 2 is the second generation of hybrid reality glasses from Microsoft, released in 2019. The first operation in mixed reality took place using the previous version of Hololens from 2016. They are controlled by voice commands and gestures and allow online access to any necessary information. Diagnostic data or three-dimensional models of organs are reproduced in the form of holograms right in front of the doctor’s eyes. So the surgeon can see all the stages of the operation and compare with what he is doing, as well as share experience with colleagues around the world: usually only a few people can fit in the operating room.
Now glasses have become more accurate and functional. The HoloLens 2 headset automatically detects where the surgeon is looking and projects the image there. Holograms can also be moved with gestures.
In addition to operations, HoloLens is used for virtual rounds, as well as for remote treatment of patients with COVID-19. The main barrier to widespread adoption so far remains the price: the glasses cost $3, and bundled with a subscription to Dynamics 500 Remote Assist – $365.
What is mixed reality?
Mixed or hybrid reality (Mixed Reality, MR) combines virtual and ordinary reality: here hologram images interact with real objects online.
The boundaries between mixed, augmented and virtual reality are not yet clearly defined. Conditionally allocate the following sequence:
How MR helps doctors
- Training
At the time of graduation, about 30% of surgeons cannot operate on their own, and their knowledge and skills are far behind the latest technologies in medicine. At the same time, the number of people who need surgery will grow from 2025 to 34 by 88, which will lead to a shortage of good surgeons.
With the help of mixed reality, beginners can see how their experienced colleagues around the world operate. In MR, three-dimensional models of the operated organs and all the necessary information are visible, and the surgeon himself comments on his actions online. Therefore, it is easier for interns and residents to perceive and remember the course of the operation. With the help of 3D models in hybrid reality, you can also train future doctors in anatomy and other disciplines, reproducing any damage and pathology.
- Working with snapshots
With the help of mixed reality, you can visualize any diagnostic images: CT, MRI, X-ray or ultrasound. By placing them in front of you in the form of a three-dimensional model and increasing as necessary, it is easier to find pathologies and make a diagnosis.
- Minimally invasive surgeries
Such interventions are often used in microvascular surgery or arthroscopy – the damaged joint is operated on using an arthroscope through a microincision. Usually, the surgeon has to control the operation with the help of video received through the microcameras inside, and carefully calibrate each movement. Volumetric holograms in mixed reality make the process much more convenient, allow you to act more accurately and minimize risks.
- General Operations
3D models of operated organs help to work out every step before the operation. During the operation, you can apply a hologram directly to the organ and consult with colleagues who connect through an online application.
- Data exchange
The mass adoption of mixed reality headsets will allow medical facilities and research institutes around the world to share information about diagnosis and treatment. This will help bring medicine to uniform standards in developed and developing countries, as well as accelerate its development.
- Telemedicine
Online consultations with VR or MR elements will be especially in demand with the development of telemedicine. The doctor will be able to connect to the treatment of the patient from anywhere, instantly receiving all the diagnostic data and examining the affected organs in the form of a three-dimensional hologram. At the same time, he does not even need a laptop or LED-monitor to view the pictures.
How VR Helps Fight Mental and Neurological Diseases
With the help of VR helmets, early diagnosis of Alzheimer’s disease is available. It accounts for 60-70% of all cases of senile dementia in the world. The technique was invented by John O’Keeffe, a professor at the University of London and a Nobel Prize winner in medicine and physiology. He found that the part of the brain that is responsible for determining the position of the body in space is the first to suffer in Alzheimer’s disease. This happens even before the first obvious signs, such as memory loss. The patient puts on a VR helmet and performs a series of tasks, after which the doctor can indicate the likelihood of developing the disease.
Similarly, multiple sclerosis is diagnosed. The VR simulator helps to identify balance problems that often signal damage to the nervous system in multiple sclerosis even before the main symptoms appear.
Finally, with the help of VR, the rehabilitation of patients with Parkinson’s disease is carried out. The simulator allows you to train balance and balance in a safe environment without the risk of falling and getting injured. At the same time, the system adapts to the patient and offers him more complex tasks in order to improve muscle tone.
How mixed reality will change surgery in the future
With the help of MR, VR and AR, experienced surgeons can help less experienced colleagues from regions where it is difficult to find a good specialist. This will make quality medicine widely available.
Microsoft also plans to collect large amounts of activity data to train HoloLens with off-the-shelf algorithms. The glasses will be able to determine by the movement of the doctor’s eyes that he is at a loss in making a decision, and prompt him to take the right action.
Subscribe also to the Trends Telegram channel and stay up to date with current trends and forecasts about the future of technology, economics, education and innovation.