Surgical-DeSAM: decoupling SAM for instrument segmentation in robotic surgery.

PURPOSE: The recent segment anything model (SAM) has demonstrated impressive performance with point, text or bounding box prompts, in various applications. However, in safety-critical surgical tasks, prompting is not possible due to (1) the lack of per-frame prompts for supervised learning, (2) it is unrealistic to prompt frame-by-frame in a real-time tracking application, and (3) it is expensive to annotate prompts for offline applications. METHODS: We develop Surgical-DeSAM to generate automatic bounding box prompts for decoupling SAM to obtain instrument segmentation in real-time robotic surgery. We utilise a commonly used detection architecture, DETR, and fine-tuned it to obtain bounding box prompt for the instruments. We then empolyed decoupling SAM (DeSAM) by replacing the image encoder with DETR encoder and fine-tune prompt encoder and mask decoder to obtain instance segmentation for the surgical instruments. To improve detection performance, we adopted the Swin-transformer to better feature representation. RESULTS: The proposed method has been validated on two publicly available datasets from the MICCAI surgical instruments segmentation challenge EndoVis 2017 and 2018. The performance of our method is also compared with SOTA instrument segmentation methods and demonstrated significant improvements with dice metrics of 89.62 and 90.70 for the EndoVis 2017 and 2018 CONCLUSION: Our extensive experiments and validations demonstrate that Surgical-DeSAM enables real-time instrument segmentation without any additional prompting and outperforms other SOTA segmentation methods.

PitSurgRT: real-time localization of critical anatomical structures in endoscopic pituitary surgery.

PURPOSE: Endoscopic pituitary surgery entails navigating through the nasal cavity and sphenoid sinus to access the sella using an endoscope. This procedure is intricate due to the proximity of crucial anatomical structures (e.g. carotid arteries and optic nerves) to pituitary tumours, and any unintended damage can lead to severe complications including blindness and death. Intraoperative guidance during this surgery could support improved localization of the critical structures leading to reducing the risk of complications. METHODS: A deep learning network PitSurgRT is proposed for real-time localization of critical structures in endoscopic pituitary surgery. The network uses high-resolution net (HRNet) as a backbone with a multi-head for jointly localizing critical anatomical structures while segmenting larger structures simultaneously. Moreover, the trained model is optimized and accelerated by using TensorRT. Finally, the model predictions are shown to neurosurgeons, to test their guidance capabilities. RESULTS: Compared with the state-of-the-art method, our model significantly reduces the mean error in landmark detection of the critical structures from 138.76 to 54.40 pixels in a 1280 × 720-pixel image. Furthermore, the semantic segmentation of the most critical structure, sella, is improved by 4.39% IoU. The inference speed of the accelerated model achieves 298 frames per second with floating-point-16 precision. In the study of 15 neurosurgeons, 88.67% of predictions are considered accurate enough for real-time guidance. CONCLUSION: The results from the quantitative evaluation, real-time acceleration, and neurosurgeon study demonstrate the proposed method is highly promising in providing real-time intraoperative guidance of the critical anatomical structures in endoscopic pituitary surgery.

The distinct roles of reinforcement learning between pre-procedure and intra-procedure planning for prostate biopsy.

PURPOSE: Magnetic resonance (MR) imaging targeted prostate cancer (PCa) biopsy enables precise sampling of MR-detected lesions, establishing its importance in recommended clinical practice. Planning for the ultrasound-guided procedure involves pre-selecting needle sampling positions. However, performing this procedure is subject to a number of factors, including MR-to-ultrasound registration, intra-procedure patient movement and soft tissue motions. When a fixed pre-procedure planning is carried out without intra-procedure adaptation, these factors will lead to sampling errors which could cause false positives and false negatives. Reinforcement learning (RL) has been proposed for procedure plannings on similar applications such as this one, because intelligent agents can be trained for both pre-procedure and intra-procedure planning. However, it is not clear if RL is beneficial when it comes to addressing these intra-procedure errors. METHODS: In this work, we develop and compare imitation learning (IL), supervised by demonstrations of predefined sampling strategy, and RL approaches, under varying degrees of intra-procedure motion and registration error, to represent sources of targeting errors likely to occur in an intra-operative procedure. RESULTS: Based on results using imaging data from 567 PCa patients, we demonstrate the efficacy and value in adopting RL algorithms to provide intelligent intra-procedure action suggestions, compared to IL-based planning supervised by commonly adopted policies. CONCLUSIONS: The improvement in biopsy sampling performance for intra-procedure planning has not been observed in experiments with only pre-procedure planning. These findings suggest a strong role for RL in future prospective studies which adopt intra-procedure planning. Our open source code implementation is available here .

Image-guidance in endoscopic pituitary surgery: an in-silico study of errors involved in tracker-based techniques.

Background: Endoscopic endonasal surgery is an established minimally invasive technique for resecting pituitary adenomas. However, understanding orientation and identifying critical neurovascular structures in this anatomically dense region can be challenging. In clinical practice, commercial navigation systems use a tracked pointer for guidance. Augmented Reality (AR) is an emerging technology used for surgical guidance. It can be tracker based or vision based, but neither is widely used in pituitary surgery. Methods: This pre-clinical study aims to assess the accuracy of tracker-based navigation systems, including those that allow for AR. Two setups were used to conduct simulations: (1) the standard pointer setup, tracked by an infrared camera; and (2) the endoscope setup that allows for AR, using reflective markers on the end of the endoscope, tracked by infrared cameras. The error sources were estimated by calculating the Euclidean distance between a point's true location and the point's location after passing it through the noisy system. A phantom study was then conducted to verify the in-silico simulation results and show a working example of image-based navigation errors in current methodologies. Results: The errors of the tracked pointer and tracked endoscope simulations were 1.7 and 2.5 mm respectively. The phantom study showed errors of 2.14 and 3.21 mm for the tracked pointer and tracked endoscope setups respectively. Discussion: In pituitary surgery, precise neighboring structure identification is crucial for success. However, our simulations reveal that the errors of tracked approaches were too large to meet the fine error margins required for pituitary surgery. In order to achieve the required accuracy, we would need much more accurate tracking, better calibration and improved registration techniques.

The value of Augmented Reality in surgery - A usability study on laparoscopic liver surgery.

Augmented Reality (AR) is considered to be a promising technology for the guidance of laparoscopic liver surgery. By overlaying pre-operative 3D information of the liver and internal blood vessels on the laparoscopic view, surgeons can better understand the location of critical structures. In an effort to enable AR, several authors have focused on the development of methods to obtain an accurate alignment between the laparoscopic video image and the pre-operative 3D data of the liver, without assessing the benefit that the resulting overlay can provide during surgery. In this paper, we present a study that aims to assess quantitatively and qualitatively the value of an AR overlay in laparoscopic surgery during a simulated surgical task on a phantom setup. We design a study where participants are asked to physically localise pre-operative tumours in a liver phantom using three image guidance conditions - a baseline condition without any image guidance, a condition where the 3D surfaces of the liver are aligned to the video and displayed on a black background, and a condition where video see-through AR is displayed on the laparoscopic video. Using data collected from a cohort of 24 participants which include 12 surgeons, we observe that compared to the baseline, AR decreases the median localisation error of surgeons on non-peripheral targets from 25.8 mm to 9.2 mm. Using subjective feedback, we also identify that AR introduces usability improvements in the surgical task and increases the perceived confidence of the users. Between the two tested displays, the majority of participants preferred to use the AR overlay instead of navigated view of the 3D surfaces on a separate screen. We conclude that AR has the potential to improve performance and decision making in laparoscopic surgery, and that improvements in overlay alignment accuracy and depth perception should be pursued in the future.

Enhanced Surgeons: Understanding the Design of Augmented Reality Instructions for Keyhole Surgery.

It is important to understand how to design AR content for surgical contexts to mitigate the risk of distracting the surgeons. In this work, we test information overlays for AR guidance during keyhole surgery. We performed a preliminary evaluation of a prototype, focusing on the effects of colour, opacity, and information representation. Our work contributes insights into the design of AR guidance in surgery settings and a foundation for future research on visualisation design for surgical AR.

The Effect of Luminance on Depth Perception in Augmented Reality Guided Laparoscopic Surgery.

Depth perception is a major issue in surgical augmented reality (AR) with limited research conducted in this scientific area. This study establishes a relationship between luminance and depth perception. This can be used to improve visualisation design for AR overlay in laparoscopic surgery, providing surgeons a more accurate perception of the anatomy intraoperatively. Two experiments were conducted to determine this relationship. First, an online study with 59 participants from the general public, and second, an in-person study with 10 surgeons as participants. We developed 2 open-source software tools utilising SciKit-Surgery libraries to enable these studies and any future research. Our findings demonstrate that the higher the relative luminance, the closer a structure is perceived to the operating camera. Furthermore, the higher the luminance contrast between the two structures, the higher the depth distance perceived. The quantitative results from both experiments are in agreement, indicating that online recruitment of the general public can be helpful in similar studies. An observation made by the surgeons from the in-person study was that the light source used in laparoscopic surgery plays a role in depth perception. This is due to its varying positioning and brightness which could affect the perception of the overlaid AR. We found that luminance directly correlates with depth perception for both surgeons and the general public, regardless of other depth cues. Future research may focus on comparing different colours used in surgical AR and using a mock operating room (OR) with varying light sources and positions.

Evaluation of a calibration rig for stereo laparoscopes.

BACKGROUND: Accurate camera and hand-eye calibration are essential to ensure high-quality results in image-guided surgery applications. The process must also be able to be undertaken by a nonexpert user in a surgical setting. PURPOSE: This work seeks to identify a suitable method for tracked stereo laparoscope calibration within theater. METHODS: A custom calibration rig, to enable rapid calibration in a surgical setting, was designed. The rig was compared against freehand calibration. Stereo reprojection, stereo reconstruction, tracked stereo reprojection, and tracked stereo reconstruction error metrics were used to evaluate calibration quality. RESULTS: Use of the calibration rig reduced mean errors: reprojection (1.47 mm [SD 0.13] vs. 3.14 mm [SD 2.11], p-value 1e-8), reconstruction (1.37 px [SD 0.10] vs. 10.10 px [SD 4.54], p-value 6e-7), and tracked reconstruction (1.38 mm [SD 0.10] vs. 12.64 mm [SD 4.34], p-value 1e-6) compared with freehand calibration. The use of a ChArUco pattern yielded slightly lower reprojection errors, while a dot grid produced lower reconstruction errors and was more robust under strong global illumination. CONCLUSION: The use of the calibration rig results in a statistically significant decrease in calibration error metrics, versus freehand calibration, and represents the preferred approach for use in the operating theater.

Software tool for visualization of a probabilistic map of the epileptogenic zone from seizure semiologies.

Around one third of epilepsies are drug-resistant. For these patients, seizures may be reduced or cured by surgically removing the epileptogenic zone (EZ), which is the portion of the brain giving rise to seizures. If noninvasive data are not sufficiently lateralizing or localizing, the EZ may need to be localized by precise implantation of intracranial electroencephalography (iEEG) electrodes. The choice of iEEG targets is influenced by clinicians' experience and personal knowledge of the literature, which leads to substantial variations in implantation strategies across different epilepsy centers. The clinical diagnostic pathway for surgical planning could be supported and standardized by an objective tool to suggest EZ locations, based on the outcomes of retrospective clinical cases reported in the literature. We present an open-source software tool that presents clinicians with an intuitive and data-driven visualization to infer the location of the symptomatogenic zone, that may overlap with the EZ. The likely EZ is represented as a probabilistic map overlaid on the patient's images, given a list of seizure semiologies observed in that specific patient. We demonstrate a case study on retrospective data from a patient treated in our unit, who underwent resective epilepsy surgery and achieved 1-year seizure freedom after surgery. The resected brain structures identified as EZ location overlapped with the regions highlighted by our tool, demonstrating its potential utility.

Automatic Classification of Simulated Breast Tomosynthesis Whole Images for the Presence of Microcalcification Clusters Using Deep CNNs.

Microcalcification clusters (MCs) are among the most important biomarkers for breast cancer, especially in cases of nonpalpable lesions. The vast majority of deep learning studies on digital breast tomosynthesis (DBT) are focused on detecting and classifying lesions, especially soft-tissue lesions, in small regions of interest previously selected. Only about 25% of the studies are specific to MCs, and all of them are based on the classification of small preselected regions. Classifying the whole image according to the presence or absence of MCs is a difficult task due to the size of MCs and all the information present in an entire image. A completely automatic and direct classification, which receives the entire image, without prior identification of any regions, is crucial for the usefulness of these techniques in a real clinical and screening environment. The main purpose of this work is to implement and evaluate the performance of convolutional neural networks (CNNs) regarding an automatic classification of a complete DBT image for the presence or absence of MCs (without any prior identification of regions). In this work, four popular deep CNNs are trained and compared with a new architecture proposed by us. The main task of these trainings was the classification of DBT cases by absence or presence of MCs. A public database of realistic simulated data was used, and the whole DBT image was taken into account as input. DBT data were considered without and with preprocessing (to study the impact of noise reduction and contrast enhancement methods on the evaluation of MCs with CNNs). The area under the receiver operating characteristic curve (AUC) was used to evaluate the performance. Very promising results were achieved with a maximum AUC of 94.19% for the GoogLeNet. The second-best AUC value was obtained with a new implemented network, CNN-a, with 91.17%. This CNN had the particularity of also being the fastest, thus becoming a very interesting model to be considered in other studies. With this work, encouraging outcomes were achieved in this regard, obtaining similar results to other studies for the detection of larger lesions such as masses. Moreover, given the difficulty of visualizing the MCs, which are often spread over several slices, this work may have an important impact on the clinical analysis of DBT images.

Cross-Modality Image Registration Using a Training-Time Privileged Third Modality.

In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI [Formula: see text]). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI [Formula: see text]) are considered easier to register to T2w due to the availability of corresponding features. We propose a learning from privileged modality algorithm, using a training-only imaging modality DWI [Formula: see text], to support the challenging multi-modality registration problems. We present experimental results based on 369 sets of 3D multiparametric MRI images from 356 prostate cancer patients and report, with statistical significance, a lowered median target registration error of 4.34 mm, when registering the holdout DWI [Formula: see text] and T2w image pairs, compared with that of 7.96 mm before registration. Results also show that the proposed learning-based registration networks enabled efficient registration with comparable or better accuracy, compared with a classical iterative algorithm and other tested learning-based methods with/without the additional modality. These compared algorithms also failed to produce any significantly improved alignment between DWI [Formula: see text] and T2w in this challenging application.

Probabilistic landscape of seizure semiology localizing values.

Semiology describes the evolution of symptoms and signs during epileptic seizures and contributes to the evaluation of individuals with focal drug-resistant epilepsy for curative resection. Semiology varies in complexity from elementary sensorimotor seizures arising from primary cortex to complex behaviours and automatisms emerging from distributed cerebral networks. Detailed semiology interpreted by expert epileptologists may point towards the likely site of seizure onset, but this process is subjective. No study has captured the variances in semiological localizing values in a data-driven manner to allow objective and probabilistic determinations of implicated networks and nodes. We curated an open data set from the epilepsy literature, in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, linking semiology to hierarchical brain localizations. A total of 11 230 data points were collected from 4643 patients across 309 articles, labelled using ground truths (postoperative seizure-freedom, concordance of imaging and neurophysiology, and/or invasive EEG) and a designation method that distinguished between semiologies arising from a predefined cortical region and descriptions of neuroanatomical localizations responsible for generating a particular semiology. This allowed us to mitigate temporal lobe publication bias by filtering studies that preselected patients based on prior knowledge of their seizure foci. Using this data set, we describe the probabilistic landscape of semiological localizing values as forest plots at the resolution of seven major brain regions: temporal, frontal, cingulate, parietal, occipital, insula, and hypothalamus, and five temporal subregions. We evaluated the intrinsic value of any one semiology over all other ictal manifestations. For example, epigastric auras implicated the temporal lobe with 83% probability when not accounting for the publication bias that favoured temporal lobe epilepsies. Unbiased results for a prior distribution of cortical localizations revised the prevalence of temporal lobe epilepsies from 66% to 44%. Therefore, knowledge about the presence of epigastric auras updates localization to the temporal lobe with an odds ratio (OR) of 2.4 [CI95% (1.9, 2.9); and specifically, mesial temporal structures OR: 2.8 (2.3, 2.9)], attesting the value of epigastric auras. As a further example, although head version is thought to implicate the frontal lobes, it did not add localizing value compared with the prior distribution of cortical localizations [OR: 0.9 (0.7, 1.2)]. Objectification of the localizing values of the 12 most common semiologies provides a complementary view of brain dysfunction to that of lesion-deficit mappings, as instead of linking brain regions to phenotypic-deficits, semiological phenotypes are linked back to brain sources. This work enables coupling of seizure propagation with ictal manifestations, and clinical support algorithms for localizing seizure phenotypes.

Phase 0 Study of Vandetanib-Eluting Radiopaque Embolics as a Preoperative Embolization Treatment in Patients with Resectable Liver Malignancies.

PURPOSE: To assess the safety and tolerability of a vandetanib-eluting radiopaque embolic (BTG-002814) for transarterial chemoembolization (TACE) in patients with resectable liver malignancies. MATERIALS AND METHODS: The VEROnA clinical trial was a first-in-human, phase 0, single-arm, window-of-opportunity study. Eligible patients were aged ≥18 years and had resectable hepatocellular carcinoma (HCC) (Child-Pugh A) or metastatic colorectal cancer (mCRC). Patients received 1 mL of BTG-002814 transarterially (containing 100 mg of vandetanib) 7-21 days prior to surgery. The primary objectives were to establish the safety and tolerability of BTG-002814 and determine the concentrations of vandetanib and the N-desmethyl vandetanib metabolite in the plasma and resected liver after treatment. Biomarker studies included circulating proangiogenic factors, perfusion computed tomography, and dynamic contrast-enhanced magnetic resonance imaging. RESULTS: Eight patients were enrolled: 2 with HCC and 6 with mCRC. There was 1 grade 3 adverse event (AE) before surgery and 18 after surgery; 6 AEs were deemed to be related to BTG-002814. Surgical resection was not delayed. Vandetanib was present in the plasma of all patients 12 days after treatment, with a mean maximum concentration of 24.3 ng/mL (standard deviation ± 13.94 ng/mL), and in resected liver tissue up to 32 days after treatment (441-404,000 ng/g). The median percentage of tumor necrosis was 92.5% (range, 5%-100%). There were no significant changes in perfusion imaging parameters after TACE. CONCLUSIONS: BTG-002814 has an acceptable safety profile in patients before surgery. The presence of vandetanib in the tumor specimens up to 32 days after treatment suggests sustained anticancer activity, while the low vandetanib levels in the plasma suggest minimal release into the systemic circulation. Further evaluation of this TACE combination is warranted in dose-finding and efficacy studies.

Deep hashing for global registration of untracked 2D laparoscopic ultrasound to CT.

PURPOSE: The registration of Laparoscopic Ultrasound (LUS) to CT can enhance the safety of laparoscopic liver surgery by providing the surgeon with awareness on the relative positioning between critical vessels and a tumour. In an effort to provide a translatable solution for this poorly constrained problem, Content-based Image Retrieval (CBIR) based on vessel information has been suggested as a method for obtaining a global coarse registration without using tracking information. However, the performance of these frameworks is limited by the use of non-generalisable handcrafted vessel features. METHODS: We propose the use of a Deep Hashing (DH) network to directly convert vessel images from both LUS and CT into fixed size hash codes. During training, these codes are learnt from a patient-specific CT scan by supplying the network with triplets of vessel images which include both a registered and a mis-registered pair. Once hash codes have been learnt, they can be used to perform registration with CBIR methods. RESULTS: We test a CBIR pipeline on 11 sequences of untracked LUS distributed across 5 clinical cases. Compared to a handcrafted feature approach, our model improves the registration success rate significantly from 48% to 61%, considering a 20 mm error as the threshold for a successful coarse registration. CONCLUSIONS: We present the first DH framework for interventional multi-modal registration tasks. The presented approach is easily generalisable to other registration problems, does not require annotated data for training, and may promote the translation of these techniques.

Large scale simulation of labeled intraoperative scenes in unity.

PURPOSE: The use of synthetic or simulated data has the potential to greatly improve the availability and volume of training data for image guided surgery and other medical applications, where access to real-life training data is limited. METHODS: By using the Unity game engine, complex intraoperative scenes can be simulated. The Unity Perception package allows for randomisation of paremeters within the scene, and automatic labelling, to make simulating large data sets a trivial operation. In this work, the approach has been prototyped for liver segmentation from laparoscopic video images. 50,000 simulated images were used to train a U-Net, without the need for any manual labelling. The use of simulated data was compared against a model trained with 950 manually labelled laparoscopic images. RESULTS: When evaluated on data from 10 separate patients, synthetic data outperformed real data in 4 out of 10 cases. Average DICE scores across the 10 cases were 0.59 (synthetic data), 0.64 (real data) and 0.75 (both synthetic and real data). CONCLUSION: Synthetic data generated using this method is able to make valid inferences on real data, with average performance slightly below models trained on real data. The use of the simulated data for pre-training boosts model performance, when compared with training on real data only.

Gesture Recognition in Robotic Surgery With Multimodal Attention.

Automatically recognising surgical gestures from surgical data is an important building block of automated activity recognition and analytics, technical skill assessment, intra-operative assistance and eventually robotic automation. The complexity of articulated instrument trajectories and the inherent variability due to surgical style and patient anatomy make analysis and fine-grained segmentation of surgical motion patterns from robot kinematics alone very difficult. Surgical video provides crucial information from the surgical site with context for the kinematic data and the interaction between the instruments and tissue. Yet sensor fusion between the robot data and surgical video stream is non-trivial because the data have different frequency, dimensions and discriminative capability. In this paper, we integrate multimodal attention mechanisms in a two-stream temporal convolutional network to compute relevance scores and weight kinematic and visual feature representations dynamically in time, aiming to aid multimodal network training and achieve effective sensor fusion. We report the results of our system on the JIGSAWS benchmark dataset and on a new in vivo dataset of suturing segments from robotic prostatectomy procedures. Our results are promising and obtain multimodal prediction sequences with higher accuracy and better temporal structure than corresponding unimodal solutions. Visualization of attention scores also gives physically interpretable insights on network understanding of strengths and weaknesses of each sensor.

Radiopaque drug-eluting embolisation beads as fiducial markers for stereotactic liver radiotherapy.

OBJECTIVE: To determine the feasibility of using radiopaque (RO) beads as direct tumour surrogates for image-guided radiotherapy (IGRT) in patients with liver tumours after transarterial chemoembolisation (TACE). METHODS: A novel vandetanib-eluting RO bead was delivered via TACE as part of a first-in-human clinical trial in patients with either hepatocellular carcinoma or liver metastases from colorectal cancer. Following TACE, patients underwent simulated radiotherapy imaging with four-dimensional computed tomography (4D-CT) and cone-beam CT (CBCT) imaging. RO beads were contoured using automated thresholding, and feasibility of matching between the simulated radiotherapy planning dataset (AVE-IP image from 4D data) and CBCT scans assessed. Additional kV, MV, helical CT and CBCT images of RO beads were obtained using an in-house phantom. Stability of RO bead position was assessed by comparing 4D-CT imaging to CT scans taken 6-20 days following TACE. RESULTS: Eight patients were treated and 4D-CT and CBCT images acquired. RO beads were visible on 4D-CT and CBCT images in all cases and matching successfully performed. Differences in centre of mass of RO beads between CBCT and simulated radiotherapy planning scans (AVE-IP dataset) were 2.0 mm mediolaterally, 1.7 mm anteroposteriorally and 3.5 mm craniocaudally. RO beads in the phantom were visible on all imaging modalities assessed. RO bead position remained stable up to 29 days post TACE. CONCLUSION: RO beads are visible on IGRT imaging modalities, showing minimal artefact. They can be used for on-set matching with CBCT and remain stable over time. ADVANCES IN KNOWLEDGE: The role of RO beads as fiducial markers for stereotactic liver radiotherapy is feasible and warrants further exploration as a combination therapy approach.

Automatic, global registration in laparoscopic liver surgery.

PURPOSE: The initial registration of a 3D pre-operative CT model to a 2D laparoscopic video image in augmented reality systems for liver surgery needs to be fast, intuitive to perform and with minimal interruptions to the surgical intervention. Several recent methods have focussed on using easily recognisable landmarks across modalities. However, these methods still need manual annotation or manual alignment. We propose a novel, fully automatic pipeline for 3D-2D global registration in laparoscopic liver interventions. METHODS: Firstly, we train a fully convolutional network for the semantic detection of liver contours in laparoscopic images. Secondly, we propose a novel contour-based global registration algorithm to estimate the camera pose without any manual input during surgery. The contours used are the anterior ridge and the silhouette of the liver. RESULTS: We show excellent generalisation of the semantic contour detection on test data from 8 clinical cases. In quantitative experiments, the proposed contour-based registration can successfully estimate a global alignment with as little as 30% of the liver surface, a visibility ratio which is characteristic of laparoscopic interventions. Moreover, the proposed pipeline showed very promising results in clinical data from 5 laparoscopic interventions. CONCLUSIONS: Our proposed automatic global registration could make augmented reality systems more intuitive and usable for surgeons and easier to translate to operating rooms. Yet, as the liver is deformed significantly during surgery, it will be very beneficial to incorporate deformation into our method for more accurate registration.

SciKit-SurgeryGlenoid, an Open Source Toolkit for Glenoid Version Measurement.

Correct understanding of the geometry of the glenoid (the socket of the shoulder joint) is key to successful planning of shoulder replacement surgery. This surgery typically involves placing an implant in the shoulder joint to restore joint function. The most relevant geometry is the glenoid version, which is the angular orientation of the glenoid surface relative to the long axis of the scapula in the axial plane. However, measuring the glenoid version is not straightforward and there are multiple measurement methods in the literature and used in commercial planning software. In this paper we introduce SciKit-SurgeryGlenoid, an open source toolkit for the measurement of glenoid version. SciKit-SurgeryGlenoid contains implementations of the 4 most frequently used glenoid version measurement algorithms enabling easy and unbiased comparison of the different techniques. We present the results of using the software on 10 sets of pre-operative CT scans taken from patients who have subsequently undergone shoulder replacement surgery. We further compare these results with those obtained from a commercial implant planning software. SciKit-SurgeryGlenoid currently requires manual segmentation of the relevant anatomical features for each method. Future work will look at automating the segmentation process to build an automatic and repeatable pipeline from CT or radiograph to quantitative glenoid version measurement.

Are fiducial registration error and target registration error correlated? SciKit-SurgeryFRED for teaching and research.

Understanding the relationship between fiducial registration error (FRE) and target registration error (TRE) is important for the correct use of interventional guidance systems. Whilst it is well established that TRE is statistically independent of FRE, system users still struggle against the intuitive assumption that a low FRE indicates a low TRE. We present the SciKit-Surgery Fiducial Registration Educational Demonstrator and describe its use. SciKit-SurgeryFRED was developed to enable remote teaching of key concepts in image registration. SciKit-SurgeryFRED also supports research into user interface design for image registration systems. SciKit-SurgeryFRED can be used to enable remote tutorials covering the statistics relevant to image guided interventions. Students are able to place fiducial markers on pre and intra-operative images and observe the effects of changes in marker geometry, marker count, and fiducial localisation error on TRE and FRE. SciKit-SurgeryFRED also calculates statistical measures for the expected values of TRE and FRE. Because many registrations can be performed quickly the students can then explore potential correlations between the different statistics. SciKit-SurgeryFRED also implements a registration based game, where participants are rewarded for complete treatment of a clinical target, whilst minimising the treatment margin. We used this game to perform a remote study on registration and simulated ablation, measuring how user performance changes depending on what error statistics are made available. The results support the assumption that knowing the exact value of target registration error leads to better treatment. Display of other statistics did not have a significant impact on the treatment performance.