LapBot-Safe Chole: validation of an artificial intelligence-powered mobile game app to teach safe cholecystectomy.

BACKGROUND: Gaming can serve as an educational tool to allow trainees to practice surgical decision-making in a low-stakes environment. LapBot is a novel free interactive mobile game application that uses artificial intelligence (AI) to provide players with feedback on safe dissection during laparoscopic cholecystectomy (LC). This study aims to provide validity evidence for this mobile game. METHODS: Trainees and surgeons participated by downloading and playing LapBot on their smartphone. Players were presented with intraoperative LC scenes and required to locate their preferred location of dissection of the hepatocystic triangle. They received immediate accuracy scores and personalized feedback using an AI algorithm ("GoNoGoNet") that identifies safe/dangerous zones of dissection. Player scores were assessed globally and across training experience using non-parametric ANOVA. Three-month questionnaires were administered to assess the educational value of LapBot. RESULTS: A total of 903 participants from 64 countries played LapBot. As game difficulty increased, average scores (p < 0.0001) and confidence levels (p < 0.0001) decreased significantly. Scores were significantly positively correlated with players' case volume (p = 0.0002) and training level (p = 0.0003). Most agreed that LapBot should be incorporated as an adjunct into training programs (64.1%), as it improved their ability to reflect critically on feedback they receive during LC (47.5%) or while watching others perform LC (57.5%). CONCLUSIONS: Serious games, such as LapBot, can be effective educational tools for deliberate practice and surgical coaching by promoting learner engagement and experiential learning. Our study demonstrates that players' scores were correlated to their level of expertise, and that after playing the game, most players perceived a significant educational value.

A usability analysis of augmented reality and haptics for surgical planning.

PURPOSE: Proper visualization and interaction with complex anatomical data can improve understanding, allowing for more intuitive surgical planning. The goal of our work was to study what the most intuitive yet practical platforms for interacting with 3D medical data are in the context of surgical planning. METHODS: We compared planning using a monitor and mouse, a monitor with a haptic device, and an augmented reality (AR) head-mounted display which uses a gesture-based interaction. To determine the most intuitive system, two user studies, one with novices and one with experts, were conducted. The studies involved planning of three scenarios: (1) heart valve repair, (2) hip tumor resection, and (3) pedicle screw placement. Task completion time, NASA Task Load Index and system-specific questionnaires were used for the evaluation. RESULTS: Both novices and experts preferred the AR system for pedicle screw placement. Novices preferred the haptic system for hip tumor planning, while experts preferred the mouse and keyboard. In the case of heart valve planning, novices preferred the AR system but there was no clear preference for experts. Both groups reported that AR provides the best spatial depth perception. CONCLUSION: The results of the user studies suggest that different surgical cases may benefit from varying interaction and visualization methods. For example, for planning surgeries with implants and instrumentations, mixed reality could provide better 3D spatial perception, whereas using landmarks for delineating specific targets may be more effective using a traditional 2D interface.

Virtual and Augmented Reality in Ventriculostomy: A Systematic Review.

BACKGROUND: Ventriculostomy, one of the most common neurosurgical procedures, involves inserting a draining catheter into the brain's ventricular system to alleviate excessive cerebrospinal fluid accumulation. Traditionally, this procedure has relied on freehand techniques guided by anatomical landmarks, which have shown a high rate of misplacement. Recent advancements in virtual reality (VR) and augmented reality (AR) technologies have opened up new possibilities in the field. This comprehensive review aims to analyze the existing literature, examine the diverse applications of VR and AR in ventriculostomy procedures, address their limitations, and propose potential future directions. METHODS: A systematic search was conducted in Web of Science and PubMed databases to identify studies employing VR and AR technologies in ventriculostomy procedures. Review papers, non-English records, studies unrelated to VR/AR technologies in ventriculostomy, and supplementary documents were excluded. In total 29 papers were included in the review. RESULTS: The development of various VR and AR systems aimed at enhancing the ventriculostomy procedure are categorized according to the Data, Visualization and View taxonomy. The study investigates the data utilized by these systems, the visualizations employed, and the virtual or augmented environments created. Furthermore, the surgical scenarios and applications of each method, as well as the validation and evaluation metrics used, are discussed. DISCUSSION: The review delves into the fundamental challenges encountered in the implementation of VR and AR systems in ventriculostomy. Additionally, potential future directions and areas for improvement are proposed, addressing the identified limitations and paving the way for further advancements in the field.

Breamy: An augmented reality mHealth prototype for surgical decision-making in breast cancer.

Breast cancer is one of the most prevalent forms of cancer, affecting approximately one in eight women during their lifetime. Deciding on breast cancer treatment, which includes the choice between surgical options, frequently demands prompt decision-making within an 8-week timeframe. However, many women lack the necessary knowledge and preparation for making informed decisions. Anxiety and unsatisfactory outcomes can result from inadequate decision-making processes, leading to decisional regret and revision surgeries. Shared decision-making and personalized decision aids have shown positive effects on patient satisfaction and treatment outcomes. Here, Breamy, a prototype mobile health application that utilizes augmented reality technology to assist breast cancer patients in making more informed decisions is introduced. Breamy provides 3D visualizations of different surgical procedures, aiming to improve confidence in surgical decision-making, reduce decisional regret, and enhance patient well-being after surgery. To determine the perception of the usefulness of Breamy, data was collected from 166 participants through an online survey. The results suggest that Breamy has the potential to reduce patients' anxiety levels and assist them in decision-making.

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room.

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

SCANED: Siamese collateral assessment network for evaluation of collaterals from ischemic damage.

This study conducts collateral evaluation from ischemic damage using a deep learning-based Siamese network, addressing the challenges associated with a small and imbalanced dataset. The collateral network provides an alternative oxygen and nutrient supply pathway in ischemic stroke cases, influencing treatment decisions. Research in this area focuses on automated collateral assessment using deep learning (DL) methods to expedite decision-making processes and enhance accuracy. Our study employed a 3D ResNet-based Siamese network, referred to as SCANED, to classify collaterals as good/intermediate or poor. Utilizing non-contrast computed tomography (NCCT) images, the network automates collateral identification and assessment by analyzing tissue degeneration around the ischemic site. Relevant features from the left/right hemispheres were extracted, and Euclidean Distance (ED) was employed for similarity measurement. Finally, dichotomized classification of good/intermediate or poor collateral is performed by SCANED using an optimal threshold derived from ROC analysis. SCANED provides a sensitivity of 0.88, a specificity of 0.63, and a weighted F1 score of 0.86 in the dichotomized classification.

Education in Laparoscopic Cholecystectomy: Design and Feasibility Study of the LapBot Safe Chole Mobile Game.

BACKGROUND:  Major bile duct injuries during laparoscopic cholecystectomy (LC), often stemming from errors in surgical judgment and visual misperception of critical anatomy, significantly impact morbidity, mortality, disability, and health care costs. OBJECTIVE:  To enhance safe LC learning, we developed an educational mobile game, LapBot Safe Chole, which uses an artificial intelligence (AI) model to provide real-time coaching and feedback, improving intraoperative decision-making. METHODS:  LapBot Safe Chole offers a free, accessible simulated learning experience with real-time AI feedback. Players engage with intraoperative LC scenarios (short video clips) and identify ideal dissection zones. After the response, users receive an accuracy score from a validated AI algorithm. The game consists of 5 levels of increasing difficulty based on the Parkland grading scale for cholecystitis. RESULTS:  Beta testing (n=29) showed score improvements with each round, with attendings and senior trainees achieving top scores faster than junior residents. Learning curves and progression distinguished candidates, with a significant association between user level and scores (P=.003). Players found LapBot enjoyable and educational. CONCLUSIONS:  LapBot Safe Chole effectively integrates safe LC principles into a fun, accessible, and educational game using AI-generated feedback. Initial beta testing supports the validity of the assessment scores and suggests high adoption and engagement potential among surgical trainees.

Deep learning for collateral evaluation in ischemic stroke with imbalanced data.

PURPOSE: Collateral evaluation is typically done using visual inspection of cerebral images and thus suffers from intra- and inter-rater variability. Large open databases of ischemic stroke patients are rare, limiting the use of deep learning methods in treatment decision-making. METHODS: We adapted a pre-trained EfficientNet B0 network through transfer learning to improve collateral evaluation using slice-based and subject-level classification. Our method uses stacking and overlapping of 2D slices from a patient's 4D computed tomography angiography (CTA) and a majority voting scheme to determine a patient's final collateral grade based on all classified 2D MIPs. Class imbalance is handled in the evaluation process by using the focal loss with class weight to penalize the majority class. RESULTS: We evaluated our method using a nine-fold cross-validation performed with 83 subjects. Mean sensitivity of 0.71, specificity of 0.84, and a weighted F1 score of 0.71 in multi-class (good, intermediate, and poor) classification were obtained. Considering treatment effect, a dichotomized decision is also made for collateral scoring of a subject based on two classes (good/intermediate and poor) which achieves a sensitivity of 0.89 and specificity of 0.96 with a weighted F1 score of 0.95. CONCLUSION: An automatic and robust collateral assessment method that mitigates the issues with the small imbalanced dataset was developed. Computer-aided evaluation of collaterals can help decision-making of ischemic stroke treatment strategy in clinical settings.

Augmented reality visualization for guidance in neurovascular surgery.

In neurovascular surgery, and in particular surgery for arteriovenous malformations (AVMs), the surgeon maps pre-operative images to the patient on the operating table to aid in vessel localization and resection. This type of spatial mapping is not trivial, is time consuming, and may be prone to error. Using augmented reality (AR) we can register the microscope/camera image of the patient to pre-operative data in order to help the surgeon better understand the topology and locations of vessels that lie below the visible surface of the cortex. In this work we describe a prototype system, developed using open source software and built with off-the-shelf hardware, for AR visualization for AVM neurosurgery. Furthermore, we consider two visualization techniques, colour-coding and chromadepth, to enhance the depth perception of vessels.

Brain Shift in Neuronavigation of Brain Tumors: An Updated Review of Intra-Operative Ultrasound Applications.

Neuronavigation using pre-operative imaging data for neurosurgical guidance is a ubiquitous tool for the planning and resection of oncologic brain disease. These systems are rendered unreliable when brain shift invalidates the patient-image registration. Our previous review in 2015, Brain shift in neuronavigation of brain tumours: A review offered a new taxonomy, classification system, and a historical perspective on the causes, measurement, and pre- and intra-operative compensation of this phenomenon. Here we present an updated review using the same taxonomy and framework, focused on the developments of intra-operative ultrasound-based brain shift research from 2015 to the present (2020). The review was performed using PubMed to identify articles since 2015 with the specific words and phrases: "Brain shift" AND "Ultrasound". Since 2015, the rate of publication of intra-operative ultrasound based articles in the context of brain shift has increased from 2-3 per year to 8-10 per year. This efficient and low-cost technology and increasing comfort among clinicians and researchers have allowed unique avenues of development. Since 2015, there has been a trend towards more mathematical advancements in the field which is often validated on publicly available datasets from early intra-operative ultrasound research, and may not give a just representation to the intra-operative imaging landscape in modern image-guided neurosurgery. Focus on vessel-based registration and virtual and augmented reality paradigms have seen traction, offering new perspectives to overcome some of the different pitfalls of ultrasound based technologies. Unfortunately, clinical adaptation and evaluation has not seen as significant of a publication boost. Brain shift continues to be a highly prevalent pitfall in maintaining accuracy throughout oncologic neurosurgical intervention and continues to be an area of active research. Intra-operative ultrasound continues to show promise as an effective, efficient, and low-cost solution for intra-operative accuracy management. A major drawback of the current research landscape is that mathematical tool validation based on retrospective data outpaces prospective clinical evaluations decreasing the strength of the evidence. The need for newer and more publicly available clinical datasets will be instrumental in more reliable validation of these methods that reflect the modern intra-operative imaging in these procedures.

Automatic collateral circulation scoring in ischemic stroke using 4D CT angiography with low-rank and sparse matrix decomposition.

PURPOSE: Sufficient collateral blood supply is crucial for favorable outcomes with endovascular treatment. The current practice of collateral scoring relies on visual inspection and thus can suffer from inter and intra-rater inconsistency. We present a robust and automatic method to score cerebral collateral blood supply to aid ischemic stroke treatment decision making. The developed method is based on 4D dynamic CT angiography (CTA) and the ASPECTS scoring protocol. METHODS: The proposed method, ACCESS (Automatic Collateral Circulation Evaluation in iSchemic Stroke), estimates a target patient's unfilled cerebrovasculature in contrast-enhanced CTA using the lack of contrast agent due to clotting. To do so, the fast robust matrix completion algorithm with in-face extended Frank-Wolfe optimization is applied on a cohort of healthy subjects and a target patient, to model the patient's unfilled vessels and the estimated full vasculature as sparse and low-rank components, respectively. The collateral score is computed as the ratio of the unfilled vessels to the full vasculature, mimicking existing clinical protocols. RESULTS: ACCESS was tested with 46 stroke patients and obtained an overall accuracy of 84.78%. The optimal threshold selection was evaluated using a receiver operating characteristics curve with the leave-one-out approach, and a mean area under the curve of 85.39% was obtained. CONCLUSION: ACCESS automates collateral scoring to mitigate the shortcomings of the standard clinical practice. It is a robust approach, which resembles how radiologists score clinical scans, and can be used to help radiologists in clinical decisions of stroke treatment.

MARIN: an open-source mobile augmented reality interactive neuronavigation system.

PURPOSE: Neuronavigation systems making use of augmented reality (AR) have been the focus of much research in the last couple of decades. In recent years, there has been considerable interest in using mobile devices for AR in the operating room (OR). We propose a complete system that performs real-time AR video augmentation on a mobile device in the context of image-guided neurosurgery. METHODS: MARIN (mobile augmented reality interactive neuronavigation system) improves upon the state of the art in terms of performance, allowing real-time augmentation, and interactivity by allowing users to interact with the displayed data. The system was tested in a user study with 17 subjects for qualitative and quantitative evaluation in the context of target localization and brought into the OR for preliminary feasibility tests, where qualitative feedback from surgeons was obtained. RESULTS: The results of the user study showed that MARIN performs significantly better in terms of both time ([Formula: see text]) and accuracy ([Formula: see text]) for the task of target localization in comparison with a traditional image-guided neurosurgery (IGNS) navigation system. Further, MARIN AR visualization was found to be more intuitive and allowed users to estimate target depth more easily. CONCLUSION: MARIN improves upon previously proposed mobile AR neuronavigation systems with its real-time performance, higher accuracy, full integration in the normal workflow and greater interactivity and customizability of the displayed information. The improvement in efficiency and usability over previous systems will facilitate bringing AR into the OR.

Augmented reality mastectomy surgical planning prototype using the HoloLens template for healthcare technology letters.

In breast reconstruction following a single mastectomy, the surgeon needs to choose between tens of available implants to find the one that can reproduce the symmetry of the patient's breasts. However, due to the lack of measurement tools this decision is made purely visually, which means the surgeon has to order multiple implants to confirm the size for every single patient. In this Letter, the authors present an augmented reality application, which enables surgeons to see the shape of the implants, as 3D holograms on the patient's body. They custom developed a two-chamber implant that can gain different shapes and be used to test the system. Furthermore, the system was tested in a user study with 13 subjects. The study showed that subjects were able to do a comparison between real and holographic implants and come to a decision about which should be used. This method can be quicker than the traditional way and eliminates sizer implants from the process. Further advantages of the method include the use of a more accurate, user-friendly device, which is easily extendable as new implants that are on the market can be easily added to the system dataset.

Cognitive load associations when utilizing auditory display within image-guided neurosurgery.

PURPOSE: The combination of data visualization and auditory display (e.g., sonification) has been shown to increase accuracy, and reduce perceived difficulty, within 3D navigation tasks. While accuracy within such tasks can be measured in real time, subjective impressions about the difficulty of a task are more elusive to obtain. Prior work utilizing electrophysiology (EEG) has found robust support that cognitive load and working memory can be monitored in real time using EEG data. METHODS: In this study, we replicated a 3D navigation task (within the context of image-guided surgery) while recording data pertaining to participants' cognitive load through the use of EEG relative alpha-band weighting data. Specifically, 13 subjects navigated a tracked surgical tool to randomly placed 3D virtual locations on a CT cerebral angiography volume while being aided by visual, aural, or both visual and aural feedback. During the study EEG data were captured from the participants, and after the study a NASA TLX questionnaire was filled out by the subjects. In addition to replicating an existing experimental design on auditory display within image-guided neurosurgery, our primary aim sought to determine whether EEG-based markers of cognitive load mirrored subjective ratings of task difficulty RESULTS : Similar to existing literature, our study found evidence consistent with the hypothesis that auditory display can increase the accuracy of navigating to a specified target. We also found significant differences in cognitive working load across different feedback modalities, but none of which supported the experiments hypotheses. Finally, we found mixed results regarding the relationship between real-time measurements of cognitive workload and a posteriori subjective impressions of task difficulty. CONCLUSIONS: Although we did not find a significant correlation between the subjective and physiological measurements, differences in cognitive working load were found. As well, our study further supports the use of auditory display in image-guided surgery.

Interaction driven enhancement of depth perception in angiographic volumes.

User interaction has the potential to greatly facilitate the exploration and understanding of 3D medical images for diagnosis and treatment. However, in certain specialized environments such as in an operating room (OR), technical and physical constraints such as the need to enforce strict sterility rules, make interaction challenging. In this paper, we propose to facilitate the intraoperative exploration of angiographic volumes by leveraging the motion of a tracked surgical pointer, a tool that is already manipulated by the surgeon when using a navigation system in the OR. We designed and implemented three interactive rendering techniques based on this principle. The benefit of each of these techniques is compared to its non-interactive counterpart in a psychophysics experiment where 20 medical imaging experts were asked to perform a reaching/targeting task while visualizing a 3D volume of angiographic data. The study showed a significant improvement of the appreciation of local vascular structure when using dynamic techniques, while not having a negative impact on the appreciation of the global structure and only a marginal impact on the execution speed. A qualitative evaluation of the different techniques showed a preference for dynamic chroma-depth in accordance with the objective metrics but a discrepancy between objective and subjective measures for dynamic aerial perspective and shading.

Gesture-based registration correction using a mobile augmented reality image-guided neurosurgery system.

In image-guided neurosurgery, a registration between the patient and their pre-operative images and the tracking of surgical tools enables GPS-like guidance to the surgeon. However, factors such as brainshift, image distortion, and registration error cause the patient-to-image alignment accuracy to degrade throughout the surgical procedure no longer providing accurate guidance. The authors present a gesture-based method for manual registration correction to extend the usage of augmented reality (AR) neuronavigation systems. The authors' method, which makes use of the touchscreen capabilities of a tablet on which the AR navigation view is presented, enables surgeons to compensate for the effects of brainshift, misregistration, or tracking errors. They tested their system in a laboratory user study with ten subjects and found that they were able to achieve a median registration RMS error of 3.51 mm on landmarks around the craniotomy of interest. This is comparable to the level of accuracy attainable with previously proposed methods and currently available commercial systems while being simpler and quicker to use. The method could enable surgeons to quickly and easily compensate for most of the observed shift. Further advantages of their method include its ease of use, its small impact on the surgical workflow and its small-time requirement.

Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: a pilot study of eight cases.

We present our work investigating the feasibility of combining intraoperative ultrasound for brain shift correction and augmented reality (AR) visualization for intraoperative interpretation of patient-specific models in image-guided neurosurgery (IGNS) of brain tumors. We combine two imaging technologies for image-guided brain tumor neurosurgery. Throughout surgical interventions, AR was used to assess different surgical strategies using three-dimensional (3-D) patient-specific models of the patient's cortex, vasculature, and lesion. Ultrasound imaging was acquired intraoperatively, and preoperative images and models were registered to the intraoperative data. The quality and reliability of the AR views were evaluated with both qualitative and quantitative metrics. A pilot study of eight patients demonstrates the feasible combination of these two technologies and their complementary features. In each case, the AR visualizations enabled the surgeon to accurately visualize the anatomy and pathology of interest for an extended period of the intervention. Inaccuracies associated with misregistration, brain shift, and AR were improved in all cases. These results demonstrate the potential of combining ultrasound-based registration with AR to become a useful tool for neurosurgeons to improve intraoperative patient-specific planning by improving the understanding of complex 3-D medical imaging data and prolonging the reliable use of IGNS.

Distance sonification in image-guided neurosurgery.

Image-guided neurosurgery, or neuronavigation, has been used to visualise the location of a surgical probe by mapping the probe location to pre-operative models of a patient's anatomy. One common limitation of this approach is that it requires the surgeon to divert their attention away from the patient and towards the neuronavigation system. In order to improve this type of application, the authors designed a system that sonifies (i.e. provides audible feedback of) distance information between a surgical probe and the location of the anatomy of interest. A user study (n = 15) was completed to determine the utility of sonified distance information within an existing neuronavigation platform (Intraoperative Brain Imaging System (IBIS) Neuronav). The authors' results were consistent with the idea that combining auditory distance cues with existing visual information from image-guided surgery systems may result in greater accuracy when locating specified points on a pre-operative scan, thereby potentially reducing the extent of the required surgical openings, as well as potentially increasing the precision of individual surgical tasks. Further, the authors' results were also consistent with the hypothesis that combining auditory and visual information reduces the perceived difficulty in locating a target location within a three-dimensional volume.