Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis.

BACKGROUND: Machine learning and robotics technologies are increasingly being used in the healthcare domain to improve the quality and efficiency of surgeries and to address challenges such as staff shortages. Robotic scrub nurses in particular offer great potential to address staff shortages by assuming nursing tasks such as the handover of surgical instruments. METHODS: We introduce a robotic scrub nurse system designed to enhance the quality of surgeries and efficiency of surgical workflows by predicting and delivering the required surgical instruments based on real-time laparoscopic video analysis. We propose a three-stage deep learning architecture consisting of a single frame-, temporal multi frame-, and informed model to anticipate surgical instruments. The anticipation model was trained on a total of 62 laparoscopic cholecystectomies. RESULTS: Here, we show that our prediction system can accurately anticipate 71.54% of the surgical instruments required during laparoscopic cholecystectomies in advance, facilitating a smoother surgical workflow and reducing the need for verbal communication. As the instruments in the left working trocar are changed less frequently and according to a standardized procedure, the prediction system works particularly well for this trocar. CONCLUSIONS: The robotic scrub nurse thus acts as a mind reader and helps to mitigate staff shortages by taking over a great share of the workload during surgeries while additionally enabling an enhanced process standardization.

Staff shortages in healthcare are an emerging problem leading to undersupply of medical experts such as scrub nurses in the operating room. The absence of these scrub nurses, who are responsible for providing surgical instruments, means that surgeries must be postponed or canceled. Robotic technologies and artificial intelligence offer great potential to address staff shortages in the operating room. We developed a robotic scrub nurse system that is able to take over the tasks of a human scrub nurse by delivering the required surgical tools. To maintain the pace of the surgery, our robotic scrub nurse system is also capable of predicting these required surgical tools in advance using artificial intelligence. The system is tested on laparoscopic cholecystectomies, a surgery, where the gallbladder is removed in a minimally invasive technique. We show that our prediction system can predict the majority of surgical instruments for this specific surgery facilitating a smoother surgical workflow and reducing the need for verbal communication. With further development, our system may help to cover the need for surgery while streamlining the surgical process through predictive support, potentially improving patient outcomes.

Towards multimodal graph neural networks for surgical instrument anticipation.

PURPOSE: Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks. METHODS: We developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task. RESULTS: Our approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations. CONCLUSION: This work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.

Extra-abdominal trocar and instrument detection for enhanced surgical workflow understanding.

PURPOSE: Video-based intra-abdominal instrument tracking for laparoscopic surgeries is a common research area. However, the tracking can only be done with instruments that are actually visible in the laparoscopic image. By using extra-abdominal cameras to detect trocars and classify their occupancy state, additional information about the instrument location, whether an instrument is still in the abdomen or not, can be obtained. This can enhance laparoscopic workflow understanding and enrich already existing intra-abdominal solutions. METHODS: A data set of four laparoscopic surgeries recorded with two time-synchronized extra-abdominal 2D cameras was generated. The preprocessed and annotated data were used to train a deep learning-based network architecture consisting of a trocar detection, a centroid tracker and a temporal model to provide the occupancy state of all trocars during the surgery. RESULTS: The trocar detection model achieves an F1 score of 95.06 ± 0.88 % . The prediction of the occupancy state yields an F1 score of 89.29 ± 5.29 % , providing a first step towards enhanced surgical workflow understanding. CONCLUSION: The current method shows promising results for the extra-abdominal tracking of trocars and their occupancy state. Future advancements include the enlargement of the data set and incorporation of intra-abdominal imaging to facilitate accurate assignment of instruments to trocars.

Audio-based event detection in the operating room.

PURPOSE: Even though workflow analysis in the operating room has come a long way, current systems are still limited to research. In the quest for a robust, universal setup, hardly any attention has been given to the dimension of audio despite its numerous advantages, such as low costs, location, and sight independence, or little required processing power. METHODOLOGY: We present an approach for audio-based event detection that solely relies on two microphones capturing the sound in the operating room. Therefore, a new data set was created with over 63 h of audio recorded and annotated at the University Hospital rechts der Isar. Sound files were labeled, preprocessed, augmented, and subsequently converted to log-mel-spectrograms that served as a visual input for an event classification using pretrained convolutional neural networks. RESULTS: Comparing multiple architectures, we were able to show that even lightweight models, such as MobileNet, can already provide promising results. Data augmentation additionally improved the classification of 11 defined classes, including inter alia different types of coagulation, operating table movements as well as an idle class. With the newly created audio data set, an overall accuracy of 90%, a precision of 91% and a F1-score of 91% were achieved, demonstrating the feasibility of an audio-based event recognition in the operating room. CONCLUSION: With this first proof of concept, we demonstrated that audio events can serve as a meaningful source of information that goes beyond spoken language and can easily be integrated into future workflow recognition pipelines using computational inexpensive architectures.

Model-based individual life-spanning documentation in visceral surgery: a proof of concept.

INTRODUCTION: Surgical documentation has many implications. However, its primary function is to transfer information about surgical procedures to other medical professionals. Thereby, written reports describing procedures in detail are the current standard, impeding comprehensive understanding of patient-individual life-spanning surgical course, especially if surgeries are performed at a timely distance and in diverse facilities. Therefore, we developed a novel model-based approach for documentation of visceral surgeries, denoted as 'Surgical Documentation Markup-Modeling' (SDM-M). MATERIAL AND METHODS: For scientific evaluation, we developed a web-based prototype software allowing for creating hierarchical anatomical models that can be modified by individual surgery-related markup information. Thus, a patient's cumulated 'surgical load' can be displayed on a timeline deploying interactive anatomical 3D models. To evaluate the possible impact on daily clinical routine, we performed an evaluation study with 24 surgeons and advanced medical students, elaborating on simulated complex surgical cases, once with classic written reports and once with our prototypical SDM-M software. RESULTS: Leveraging SDM-M in an experimental environment reduced the time needed for elaborating simulated complex surgical cases from 354 ± 85 s with the classic approach to 277 ± 128 s. (p = 0.00109) The perceived task load measured by the Raw NASA-TLX was reduced significantly (p = 0.00003) with decreased mental (p = 0.00004) and physical (p = 0.01403) demand. Also, time demand (p = 0.00041), performance (p = 0.00161), effort (p = 0.00024), and frustration (p = 0.00031) were improved significantly. DISCUSSION: Model-based approaches for life-spanning surgical documentation could improve the daily clinical elaboration and understanding of complex cases in visceral surgery. Besides reduced workload and time sparing, even a more structured assessment of individual surgical cases could foster improved planning of further surgeries, information transfer, and even scientific evaluation, considering the cumulative 'surgical load.' CONCLUSION: Life-spanning model-based documentation of visceral surgical cases could significantly improve surgery and workload.

Prospective clinical cohort study: low incidence of Barrett esophagus but high rate of reflux disease at 5-year follow-up after sleeve gastrectomy versus Roux-en-Y gastric bypass.

BACKGROUND: Gastroesophageal reflux disease seems more frequent after laparoscopic sleeve gastrectomy (LSG) than Roux-en-Y gastric bypass (LRYGB). Retrospective case series have raised concerns about a high incidence of Barrett esophagus (BE) after LSG. OBJECTIVE: This prospective clinical cohort study compared the incidence of BE ≥5 years after LSG and LRYGB. SETTING: St. Clara Hospital, Basel, and University Hospital, Zürich, Switzerland. METHODS: Patients were recruited from 2 bariatric centers where preoperative gastroscopy is standard practice and LRYGB is preferred for patients with preexisting gastroesophageal reflux disease. At follow-up ≥5 years after surgery, patients underwent gastroscopy with quadrantic biopsies from the squamocolumnar junction and metaplastic segment. Symptoms were assessed using validated questionnaires. Wireless pH measurement assessed esophageal acid exposure. RESULTS: A total of 169 patients were included, with a median 7.0 ± 1.5 years after surgery. In the LSG group (n = 83), 3 patients had endoscopically and histologically confirmed de novo BE; in the LRYGB group (n = 86), there were 2 patients with BE, 1 de novo and 1 preexisting (de novo BE, 3.6% versus 1.2%; P = .362). At follow-up, reflux symptoms were reported more frequently by the LSG group than by the LRYGB group (51.9% versus 10.5%). Similarly, moderate-to-severe reflux esophagitis (Los Angeles grade B-D) was more common (27.7% versus 5.8%) despite greater use of proton pump inhibitors (49.4% versus 19.7%), and pathologic acid exposure was more frequent in patients who underwent LSG than in patients who underwent LRYGB. CONCLUSIONS: After at least 5 years of follow-up, a higher incidence of reflux symptoms, reflux esophagitis, and pathologic esophageal acid exposure was found in patients who underwent LSG compared with patients who underwent LRYGB. However, the incidence of BE after LSG was low and not significantly different between the 2 groups.

The impact of prior endoscopic or surgical therapy on open Zenker's diverticulum surgery: analysis on a large single center cohort : Comparison of primary and revisional open surgery for Zenker's diverticulum.

BACKGROUND: Endoscopic treatment of Zenker's diverticulum is an attractive minimally invasive alternative compared to the classic open approach. However, increased recurrence rate were reported. In case of relapse, endoscopic therapy might be repeated, or alternatively open surgery is performed. This study aimed to identify potential differences in the outcomes between primary or secondary surgical treatment in Zenker's diverticulum. METHODS: From January 2003 to April 2019, 227 subsequent patients underwent surgical diverticulectomy and cervical myotomy at the surgical department of TUM. 41 of 227 patients had received previous therapy, either open or endoscopic. Perioperative parameters in priorly untreated patients were retrospectively compared to those after previous therapy (mostly endoscopic) with special regard to perioperative data and postoperative complications. Univariate and multivariate regression analyses were performed to identify predictors for postoperative complications. RESULTS: We could show that the number of complications (p = 0.047) in pretreated patients is significant higher as well as the severity after Clavien-Dindo (p = 0.025). Stapler line leakage, wound infections, and operative revision rate was higher also pretreated group. Pretreatment and surgery time showed a significant association with postoperative complications in univariate analysis. In multivariate analysis, pretreatment remained a significant independent predictor of complications. CONCLUSION: The present data indicate that endoscopic therapy might represent a risk factor for postoperative complications in case of relapse surgery. Therefore primary open surgery should be debated in patients with an increased high risk of relapse.

Long-term outcome of patients with epiphrenic diverticula: A retrospective single-center analysis over 20 years.

BACKGROUND: Epiphrenic diverticula are extremely rare. Evidence-based treatment recommendations are scarce. The primary study outcome was to examine whether surgical treatment in patients with epiphrenic diverticula leads to improved quality of life by outweighing the perioperative risks compared with conservative treatment. METHODS: All patients with an epiphrenic diverticula at our institution between 2001 and 2021 were retrospectively reviewed and followed-up using a specific questionnaire, including the Eating Assessment Tool, and Gastrointestinal Quality of Life Index quality-of-life scores. RESULTS: Of 51 patients with epiphrenic diverticula, 28 had surgery and 23 had conservative treatment. The most common symptom at presentation was dysphagia. Although 16 patients underwent open surgery, 12 had minimally invasive procedures. A prophylactic stent was applied intraoperatively in 6 patients. The morbidity rate in surgically treated patients was 50% (14/28), with a leakage in 43% (12/28; 33% for prophylactic stenting). Mortality was nil. At a median follow-up of 139 months, patients with surgery had better outcomes than those without (ie, less dysphagia [6/12 vs 11/12; P = .025]), a less likely pathologic Eating Assessment Tool score (4/12 vs 9/12; P = .041), and a nonsignificant better Gastrointestinal Quality of Life Index score (122 vs 112; P = .929). The rate of recurrence/progression of symptoms was significantly higher for conservatively treated patients (11/18 vs 6/27 for any surgery; P = .008), as well as for patients with minimally invasive procedure (5/12 vs 1/15 for open surgery; P = .030). CONCLUSION: Despite the high perioperative morbidity, surgical treatment of epiphrenic diverticula leads to an improved long-term quality of life and lower recurrence rates than conservative treatment.

Robotic Endoscope Control - State of the Art of Voice Control and Other Options for Laparoscopic Camera Robot Guidance.

In video-based surgery (VBS), the surgeon is no longer the master of his or her view. Visualization of the surgical field is mediated by a video camera guided by an assistant. The separation of visualization and surgical activities leads to significant drawbacks. Early in the history of VBS, active camera holders controlled directly by the surgeon were introduced to replace the human camera assistant. However, despite remarkable technological advances over the past 25 years, robotic camera guidance systems (RCGS) still play only a marginal role. This is less due to inadequate mechanical designs or kinematics, and mainly due to inadequate human-machine interaction. In most cases, a simple task-shift is required that increases the surgeon's mental workload instead of reducing it. The current state of the art in robotic camera guidance, as represented by the SOLOASSIST II RCGS (AKTORmed, Neutraubling, Germany), includes a sophisticated combination of direct manual control, joystick navigation and high-quality voice control that provides the most intuitive surgeon-machine interaction currently available. An in-depth analysis of further clinical needs and promising developments in operating room integration suggests that the addition of context sensitivity could actually enhance the assistive effect of RCGS, making them potentially superior to the average human camera guidance.

Systematic Review and Meta-Analysis on Colorectal Anastomotic Techniques.

Purpose: Anastomosis creation after resective gastrointestinal surgery is a crucial task. The present review examines the techniques and implants currently available for anastomosis creation and analyses to which extent they already address our clinical needs, with a special focus on their potential to enable further trauma minimization in visceral surgery. Methods: A multi-database research was conducted in MEDLINE, Scopus, and Cochrane Library. Comparative controlled and uncontrolled clinical trials dealing with anastomosis creation techniques in the intestinal tract in both German and English were included and statistically significant differences in postoperative complication incidences were assessed using the RevMan5.4 Review Manager (Cochrane Collaboration, Oxford, UK). Results: All methods and implant types were analyzed and compared with respect to four dimensions, assessing the techniques' current performances and further potentials for surgical trauma reduction. Postoperative outcome measures, such as leakage, stenosis, reoperation and mortality rates, as well as the tendency to cause bleeding, wound infections, abscesses, anastomotic hemorrhages, pulmonary embolisms, and fistulas were assessed, revealing the only statistically significant superiority of hand-suture over stapling anastomoses with respect to the occurrence of obstructions. Conclusion: Based on the overall complication rates, it is concluded that none of the anastomosis systems addresses the demands of operative trauma minimization sufficiently yet. Major problems are furthermore either low standardization potentials due to dependence on the surgeons' levels of experience, high force application requirements for the actual anastomosis creation, or large and rigid device designs interfering with flexibility demands and size restrictions of the body's natural access routes. There is still a need for innovative technologies, especially with regard to enabling incisionless interventions.