Advancing surgical VQA with scene graph knowledge.

PURPOSE: The modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with natural language capabilities is emerging as a necessity. Our work aims to advance visual question answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question-condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design. METHODS: First, we propose a surgical scene graph-based dataset, SSG-VQA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. We then propose SSG-VQA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module, which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features. RESULTS: Our comprehensive analysis shows that our SSG-VQA dataset provides a more complex, diverse, geometrically grounded, unbiased and surgical action-oriented dataset compared to existing surgical VQA datasets and SSG-VQA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries. CONCLUSION: We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. We point out that the bottleneck of the current surgical visual question-answer model lies in learning the encoded representation rather than decoding the sequence. Our SSG-VQA dataset provides a diagnostic benchmark to test the scene understanding and reasoning capabilities of the model. The source code and the dataset will be made publicly available at: https://github.com/CAMMA-public/SSG-VQA .

Dissecting self-supervised learning methods for surgical computer vision.

The field of surgical computer vision has undergone considerable breakthroughs in recent years with the rising popularity of deep neural network-based methods. However, standard fully-supervised approaches for training such models require vast amounts of annotated data, imposing a prohibitively high cost; especially in the clinical domain. Self-Supervised Learning (SSL) methods, which have begun to gain traction in the general computer vision community, represent a potential solution to these annotation costs, allowing to learn useful representations from only unlabeled data. Still, the effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. In this work, we address this critical need by investigating four state-of-the-art SSL methods (MoCo v2, SimCLR, DINO, SwAV) in the context of surgical computer vision. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection. We examine their parameterization, then their behavior with respect to training data quantities in semi-supervised settings. Correct transfer of these methods to surgery, as described and conducted in this work, leads to substantial performance gains over generic uses of SSL - up to 7.4% on phase recognition and 20% on tool presence detection - as well as state-of-the-art semi-supervised phase recognition approaches by up to 14%. Further results obtained on a highly diverse selection of surgical datasets exhibit strong generalization properties. The code is available at https://github.com/CAMMA-public/SelfSupSurg.

Face detection in the operating room: comparison of state-of-the-art methods and a self-supervised approach.

PURPOSE: Face detection is a needed component for the automatic analysis and assistance of human activities during surgical procedures. Efficient face detection algorithms can indeed help to detect and identify the persons present in the room and also be used to automatically anonymize the data. However, current algorithms trained on natural images do not generalize well to the operating room (OR) images. In this work, we provide a comparison of state-of-the-art face detectors on OR data and also present an approach to train a face detector for the OR by exploiting non-annotated OR images. METHODS: We propose a comparison of six state-of-the-art face detectors on clinical data using multi-view OR faces, a dataset of OR images capturing real surgical activities. We then propose to use self-supervision, a domain adaptation method, for the task of face detection in the OR. The approach makes use of non-annotated images to fine-tune a state-of-the-art detector for the OR without using any human supervision. RESULTS: The results show that the best model, namely the tiny face detector, yields an average precision of 0.556 at intersection over union of 0.5. Our self-supervised model using non-annotated clinical data outperforms this result by 9.2%. CONCLUSION: We present the first comparison of state-of-the-art face detectors on OR images and show that results can be significantly improved by using self-supervision on non-annotated data.

Design and Validation of an Open-Source, Partial Task Trainer for Endonasal Neuro-Endoscopic Skills Development: Indian Experience.

BACKGROUND: Box trainers are ideal simulators, given they are inexpensive, accessible, and use appropriate fidelity. OBJECTIVE: The development and validation of an open-source, partial task simulator that teaches the fundamental skills necessary for endonasal skull-base neuro-endoscopic surgery. METHODS: We defined the Neuro-Endo-Trainer (NET) SkullBase-Task-GraspPickPlace with an activity area by analyzing the computed tomography scans of 15 adult patients with sellar suprasellar parasellar tumors. Four groups of participants (Group E, n = 4: expert neuroendoscopists; Group N, n =19: novice neurosurgeons; Group R, n = 11: neurosurgery residents with multiple iterations; and Group T, n = 27: neurosurgery residents with single iteration) performed grasp, pick, and place tasks using NET and were graded on task completion time and skills assessment scale score. RESULTS: Group E had lower task completion times and greater skills assessment scale scores than both Group N and R (P ≤ 0.03, 0.001). The performance of Groups N and R was found to be equivalent; in self-assessing neuro-endoscopic skill, the participants in these groups were found to have equally low pretraining scores (4/10) with significant improvement shown after NET simulation (6, 7 respectively). Angled scopes resulted in decreased scores with tilted plates compared with straight plates (30° P ≤ 0.04, 45° P ≤ 0.001). With tilted plates, decreased scores were observed when we compared the 0° with 45° endoscope (right, P ≤ 0.008; left, P ≤ 0.002). CONCLUSIONS: The NET, a face and construct valid open-source partial task neuroendoscopic trainer, was designed. Presimulation novice neurosurgeons and neurosurgical residents were described as having insufficient skills and preparation to practice neuro-endoscopy. Plate tilt and endoscope angle were shown to be important factors in participant performance. The NET was found to be a useful partial-task trainer for skill building in neuro-endoscopy.

Quantitative analysis of the Kawase versus the modified Dolenc-Kawase approach for middle cranial fossa lesions with variable anteroposterior extension.

OBJECT: The surgical corridor to the upper third of the clivus and ventral brainstem is hindered by critical neurovascular structures, such as the cavernous sinus, petrous apex, and tentorium. The traditional Kawase approach provides a 10 × 5-mm fenestration at the petrous apex of the temporal bone between the 5th cranial nerve and internal auditory canal. Due to interindividual variability, sometimes this area proves to be insufficient as a corridor to the posterior cranial fossa. The authors describe a modification to the technique of the extradural anterior petrosectomy consisting of additional transcavernous exploration and medial mobilization of the cisternal component of the trigeminal nerve. This approach is termed the modified Dolenc-Kawase (MDK) approach. METHODS: The authors describe a volumetric analysis of temporal bones with 3D laser scanning of dry and drilled bones for respective triangles and rhomboid areas, and they compare the difference of exposure with traditional versus modified approaches on cadaver dissection. Twelve dry temporal bones were laser scanned, and mesh-based volumetric analysis was done followed by drilling of the Kawase triangle and MDK rhomboid. Five cadaveric heads were drilled on alternate sides with both approaches for evaluation of the area exposed, surgical freedom, and angle of approach. RESULTS: The MDK approach provides an approximately 1.5 times larger area and 2.0 times greater volume of bone at the anterior petrous apex compared with the Kawase's approach. Cadaver dissection objectified the technical feasibility of the MDK approach, providing nearly 1.5-2 times larger fenestration with improved view and angulation to the posterior cranial fossa. Practical application in 6 patients with different lesions proves clinical applicability of the MDK approach. CONCLUSIONS: The larger fenestration at the petrous apex achieved with the MDK approach provides greater surgical freedom at the Dorello canal, gasserian ganglion, and prepontine area and better anteroposterior angulation than the traditional Kawase approach. Additional anterior clinoidectomy and transcavernous exposure helps in dealing with basilar artery aneurysms.

Quantitative analysis of variable extent of anterior clinoidectomy with intradural and extradural approaches: 3-dimensional analysis and cadaver dissection.

BACKGROUND: Drilling of the anterior clinoid process (ACP) is an integral component of surgical approaches for central and paracentral skull base lesions. The technique to drill ACP has evolved from pure intradural to extradural and combined techniques. OBJECTIVE: To describe the computerized morphometric evaluation of exposure of optic nerve and internal carotid artery with proposed tailored intradural (IDAC) and complete extradural (EDAC) anterior clinoidectomy. METHODS: We describe a morphometric subdivision of ACP into 4 quadrangles and 1 triangle on the basis of fixed bony landmarks. Computerized volumetric analysis with 3-dimensional laser scanning of dry-drilled bones for respective tailored IDAC and EDAC was performed. Both approaches were compared for the area and length of the optic nerve and internal carotid artery. Five cadaver heads were dissected on alternate sides with intradural and extradural techniques to evaluate exposure, surgical freedom, and angulation of approach. RESULTS: Complete anterior clinoidectomy provides a 2.5-times larger area and 2.7-times larger volume of ACP. Complete clinoidectomy deroofed the optic nerve to an equal extent as by proposed the partial tailored clinoidectomy approach. Tailored IDAC exposes only the distal dural ring, whereas complete EDAC exposes both the proximal and distal dural rings with complete exposure of the carotid cave. CONCLUSION: Quantitative comparative evaluation provides details of exposure and surgical ease with both techniques. We promote hybrid/EDAC technique for vascular pathologies because of better anatomic orientation. Extradural clinoidectomy is the preferred technique for midline cranial neoplasia. An awareness of different variations of clinoidectomy can prevent dependency on any particular approach and facilitate flexibility.