A Deep Learning-Based Camera Approach for Vital Sign Monitoring Using Thermography Images for ICU Patients.

Infrared thermography for camera-based skin temperature measurement is increasingly used in medical practice, e.g., to detect fevers and infections, such as recently in the COVID-19 pandemic. This contactless method is a promising technology to continuously monitor the vital signs of patients in clinical environments. In this study, we investigated both skin temperature trend measurement and the extraction of respiration-related chest movements to determine the respiratory rate using low-cost hardware in combination with advanced algorithms. In addition, the frequency of medical examinations or visits to the patients was extracted. We implemented a deep learning-based algorithm for real-time vital sign extraction from thermography images. A clinical trial was conducted to record data from patients on an intensive care unit. The YOLOv4-Tiny object detector was applied to extract image regions containing vital signs (head and chest). The infrared frames were manually labeled for evaluation. Validation was performed on a hold-out test dataset of 6 patients and revealed good detector performance (0.75 intersection over union, 0.94 mean average precision). An optical flow algorithm was used to extract the respiratory rate from the chest region. The results show a mean absolute error of 2.69 bpm. We observed a computational performance of 47 fps on an NVIDIA Jetson Xavier NX module for YOLOv4-Tiny, which proves real-time capability on an embedded GPU system. In conclusion, the proposed method can perform real-time vital sign extraction on a low-cost system-on-module and may thus be a useful method for future contactless vital sign measurements.

Towards multimodal graph neural networks for surgical instrument anticipation.

PURPOSE: Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks. METHODS: We developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task. RESULTS: Our approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations. CONCLUSION: This work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.

Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis.

BACKGROUND: Machine learning and robotics technologies are increasingly being used in the healthcare domain to improve the quality and efficiency of surgeries and to address challenges such as staff shortages. Robotic scrub nurses in particular offer great potential to address staff shortages by assuming nursing tasks such as the handover of surgical instruments. METHODS: We introduce a robotic scrub nurse system designed to enhance the quality of surgeries and efficiency of surgical workflows by predicting and delivering the required surgical instruments based on real-time laparoscopic video analysis. We propose a three-stage deep learning architecture consisting of a single frame-, temporal multi frame-, and informed model to anticipate surgical instruments. The anticipation model was trained on a total of 62 laparoscopic cholecystectomies. RESULTS: Here, we show that our prediction system can accurately anticipate 71.54% of the surgical instruments required during laparoscopic cholecystectomies in advance, facilitating a smoother surgical workflow and reducing the need for verbal communication. As the instruments in the left working trocar are changed less frequently and according to a standardized procedure, the prediction system works particularly well for this trocar. CONCLUSIONS: The robotic scrub nurse thus acts as a mind reader and helps to mitigate staff shortages by taking over a great share of the workload during surgeries while additionally enabling an enhanced process standardization.

Staff shortages in healthcare are an emerging problem leading to undersupply of medical experts such as scrub nurses in the operating room. The absence of these scrub nurses, who are responsible for providing surgical instruments, means that surgeries must be postponed or canceled. Robotic technologies and artificial intelligence offer great potential to address staff shortages in the operating room. We developed a robotic scrub nurse system that is able to take over the tasks of a human scrub nurse by delivering the required surgical tools. To maintain the pace of the surgery, our robotic scrub nurse system is also capable of predicting these required surgical tools in advance using artificial intelligence. The system is tested on laparoscopic cholecystectomies, a surgery, where the gallbladder is removed in a minimally invasive technique. We show that our prediction system can predict the majority of surgical instruments for this specific surgery facilitating a smoother surgical workflow and reducing the need for verbal communication. With further development, our system may help to cover the need for surgery while streamlining the surgical process through predictive support, potentially improving patient outcomes.

Prediction of Serious Adverse Events from Nighttime Vital Signs Values.

The period directly following surgery is critical for patients as they are at risk of infections and other types of complications, often summarized as severe adverse events (SAE). We hypothesize that impending complications might alter the circadian rhythm and, therefore, be detectable during the night before. We propose a SMOTE-enhanced XGBoost prediction model that classifies nighttime vital signs depending on whether they precede a serious adverse event or come from a patient that does not have a complication at all, based on data from 450 postoperative patients. The approach showed respectable results, producing a ROC-AUC score of 0.65 and an accuracy of 0.75. These findings demonstrate the need for further investigation.