Surgical data science - from concepts toward clinical translation.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

An automatic framework for fusing information from differently stained consecutive digital whole slide images: A case study in renal histology.

OBJECTIVE: This article presents an automatic image processing framework to extract quantitative high-level information describing the micro-environment of glomeruli in consecutive whole slide images (WSIs) processed with different staining modalities of patients with chronic kidney rejection after kidney transplantation. METHODS: This four-step framework consists of: 1) approximate rigid registration, 2) cell and anatomical structure segmentation 3) fusion of information from different stainings using a newly developed registration algorithm 4) feature extraction. RESULTS: Each step of the framework is validated independently both quantitatively and qualitatively by pathologists. An illustration of the different types of features that can be extracted is presented. CONCLUSION: The proposed generic framework allows for the analysis of the micro-environment surrounding large structures that can be segmented (either manually or automatically). It is independent of the segmentation approach and is therefore applicable to a variety of biomedical research questions. SIGNIFICANCE: Chronic tissue remodelling processes after kidney transplantation can result in interstitial fibrosis and tubular atrophy (IFTA) and glomerulosclerosis. This pipeline provides tools to quantitatively analyse, in the same spatial context, information from different consecutive WSIs and help researchers understand the complex underlying mechanisms leading to IFTA and glomerulosclerosis.

Graph-based description of tertiary lymphoid organs at single-cell level.

Our aim is to complement observer-dependent approaches of immune cell evaluation in microscopy images with reproducible measures for spatial composition of lymphocytic infiltrates. Analyzing such patterns of inflammation is becoming increasingly important for therapeutic decisions, for example in transplantation medicine or cancer immunology. We developed a graph-based assessment of lymphocyte clustering in full whole slide images. Based on cell coordinates detected in the full image, a Delaunay triangulation and distance criteria are used to build neighborhood graphs. The composition of nodes and edges are used for classification, e.g. using a support vector machine. We describe the variability of these infiltrates on CD3/CD20 duplex staining in renal biopsies of long-term functioning allografts, in breast cancer cases, and in lung tissue of cystic fibrosis patients. The assessment includes automated cell detection, identification of regions of interest, and classification of lymphocytic clusters according to their degree of organization. We propose a neighborhood feature which considers the occurrence of edges with a certain type in the graph to distinguish between phenotypically different immune infiltrates. Our work addresses a medical need and provides a scalable framework that can be easily adjusted to the requirements of different research questions.

Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks.

PURPOSE: Manual feedback from senior surgeons observing less experienced trainees is a laborious task that is very expensive, time-consuming and prone to subjectivity. With the number of surgical procedures increasing annually, there is an unprecedented need to provide an accurate, objective and automatic evaluation of trainees' surgical skills in order to improve surgical practice. METHODS: In this paper, we designed a convolutional neural network (CNN) to classify surgical skills by extracting latent patterns in the trainees' motions performed during robotic surgery. The method is validated on the JIGSAWS dataset for two surgical skills evaluation tasks: classification and regression. RESULTS: Our results show that deep neural networks constitute robust machine learning models that are able to reach new competitive state-of-the-art performance on the JIGSAWS dataset. While we leveraged from CNNs' efficiency, we were able to minimize its black-box effect using the class activation map technique. CONCLUSIONS: This characteristic allowed our method to automatically pinpoint which parts of the surgery influenced the skill evaluation the most, thus allowing us to explain a surgical skill classification and provide surgeons with a novel personalized feedback technique. We believe this type of interpretable machine learning model could integrate within "Operation Room 2.0" and support novice surgeons in improving their skills to eventually become experts.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students.

Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this paper investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. Seventy six medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: 1) labeling of images showing tissue regions and 2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.

Surgical motion analysis using discriminative interpretable patterns.

OBJECTIVE: The analysis of surgical motion has received a growing interest with the development of devices allowing their automatic capture. In this context, the use of advanced surgical training systems makes an automated assessment of surgical trainee possible. Automatic and quantitative evaluation of surgical skills is a very important step in improving surgical patient care. MATERIAL AND METHOD: In this paper, we present an approach for the discovery and ranking of discriminative and interpretable patterns of surgical practice from recordings of surgical motions. A pattern is defined as a series of actions or events in the kinematic data that together are distinctive of a specific gesture or skill level. Our approach is based on the decomposition of continuous kinematic data into a set of overlapping gestures represented by strings (bag of words) for which we compute comparative numerical statistic (tf-idf) enabling the discriminative gesture discovery via its relative occurrence frequency. RESULTS: We carried out experiments on three surgical motion datasets. The results show that the patterns identified by the proposed method can be used to accurately classify individual gestures, skill levels and surgical interfaces. We also present how the patterns provide a detailed feedback on the trainee skill assessment. CONCLUSIONS: The proposed approach is an interesting addition to existing learning tools for surgery as it provides a way to obtain a feedback on which parts of an exercise have been used to classify the attempt as correct or incorrect.

Toward a standard ontology of surgical process models.

PURPOSE: The development of common ontologies has recently been identified as one of the key challenges in the emerging field of surgical data science (SDS). However, past and existing initiatives in the domain of surgery have mainly been focussing on individual groups and failed to achieve widespread international acceptance by the research community. To address this challenge, the authors of this paper launched a European initiative-OntoSPM Collaborative Action-with the goal of establishing a framework for joint development of ontologies in the field of SDS. This manuscript summarizes the goals and the current status of the international initiative. METHODS: A workshop was organized in 2016, gathering the main European research groups having experience in developing and using ontologies in this domain. It led to the conclusion that a common ontology for surgical process models (SPM) was absolutely needed, and that the existing OntoSPM ontology could provide a good starting point toward the collaborative design and promotion of common, standard ontologies on SPM. RESULTS: The workshop led to the OntoSPM Collaborative Action-launched in mid-2016-with the objective to develop, maintain and promote the use of common ontologies of SPM relevant to the whole domain of SDS. The fundamental concept, the architecture, the management and curation of the common ontology have been established, making it ready for wider public use. CONCLUSION: The OntoSPM Collaborative Action has been in operation for 24 months, with a growing dedicated membership. Its main result is a modular ontology, undergoing constant updates and extensions, based on the experts' suggestions. It remains an open collaborative action, which always welcomes new contributors and applications.

Sequential surgical signatures in micro-suturing task.

PURPOSE: Surgical processes are generally only studied by identifying differences in populations such as participants or level of expertise. But the similarity between this population is also important in understanding the process. We therefore proposed to study these two aspects. METHODS: In this article, we show how similarities in process workflow within a population can be identified as sequential surgical signatures. To this purpose, we have proposed a pattern mining approach to identify these signatures. VALIDATION: We validated our method with a data set composed of seventeen micro-surgical suturing tasks performed by four participants with two levels of expertise. RESULTS: We identified sequential surgical signatures specific to each participant, shared between participants with and without the same level of expertise. These signatures are also able to perfectly define the level of expertise of the participant who performed a new micro-surgical suturing task. However, it is more complicated to determine who the participant is, and the method correctly determines this information in only 64% of cases. CONCLUSION: We show for the first time the concept of sequential surgical signature. This new concept has the potential to further help to understand surgical procedures and provide useful knowledge to define future CAS systems.

Surgical skills: Can learning curves be computed from recordings of surgical activities?

PURPOSE: Surgery is one of the riskiest and most important medical acts that are performed today. The need to improve patient outcomes and surgeon training, and to reduce the costs of surgery, has motivated the equipment of operating rooms with sensors that record surgical interventions. The richness and complexity of the data that are collected call for new methods to support computer-assisted surgery. The aim of this paper is to support the monitoring of junior surgeons learning their surgical skill sets. METHODS: Our method is fully automatic and takes as input a series of surgical interventions each represented by a low-level recording of all activities performed by the surgeon during the intervention (e.g., cut the skin with a scalpel). Our method produces a curve describing the process of standardization of the behavior of junior surgeons. Given the fact that junior surgeons receive constant feedback from senior surgeons during surgery, these curves can be directly interpreted as learning curves. RESULTS: Our method is assessed using the behavior of a junior surgeon in anterior cervical discectomy and fusion surgery over his first three years after residency. They revealed the ability of the method to accurately represent the surgical skill evolution. We also showed that the learning curves can be computed by phases allowing a finer evaluation of the skill progression. CONCLUSION: Preliminary results suggest that our approach constitutes a useful addition to surgical training monitoring.

Evaluation of contactless human-machine interface for robotic surgical training.

PURPOSE: Teleoperated robotic systems are nowadays routinely used for specific interventions. Benefits of robotic training courses have already been acknowledged by the community since manipulation of such systems requires dedicated training. However, robotic surgical simulators remain expensive and require a dedicated human-machine interface. METHODS: We present a low-cost contactless optical sensor, the Leap Motion, as a novel control device to manipulate the RAVEN-II robot. We compare peg manipulations during a training task with a contact-based device, the electro-mechanical Sigma.7. We perform two complementary analyses to quantitatively assess the performance of each control method: a metric-based comparison and a novel unsupervised spatiotemporal trajectory clustering. RESULTS: We show that contactless control does not offer as good manipulability as the contact-based. Where part of the metric-based evaluation presents the mechanical control better than the contactless one, the unsupervised spatiotemporal trajectory clustering from the surgical tool motions highlights specific signature inferred by the human-machine interfaces. CONCLUSIONS: Even if the current implementation of contactless control does not overtake manipulation with high-standard mechanical interface, we demonstrate that using the optical sensor complete control of the surgical instruments is feasible. The proposed method allows fine tracking of the trainee's hands in order to execute dexterous laparoscopic training gestures. This work is promising for development of future human-machine interfaces dedicated to robotic surgical training systems.

Finding discriminative and interpretable patterns in sequences of surgical activities.

OBJECTIVE: Surgery is one of the riskiest and most important medical acts that is performed today. Understanding the ways in which surgeries are similar or different from each other is of major interest to understand and analyze surgical behaviors. This article addresses the issue of identifying discriminative patterns of surgical practice from recordings of surgeries. These recordings are sequences of low-level surgical activities representing the actions performed by surgeons during surgeries. MATERIALS AND METHOD: To discover patterns that are specific to a group of surgeries, we use the vector space model (VSM) which is originally an algebraic model for representing text documents. We split long sequences of surgical activities into subsequences of consecutive activities. We then compute the relative frequencies of these subsequences using the tf*idf framework and we use the Cosine similarity to classify the sequences. This process makes it possible to discover which patterns discriminate one set of surgeries recordings from another set. RESULTS: Experiments were performed on 40 neurosurgeries of anterior cervical discectomy (ACD). The results demonstrate that our method accurately identifies patterns that can discriminate between (1) locations where the surgery took place, (2) levels of expertise of surgeons (i.e., expert vs. intermediate) and even (3) individual surgeons who performed the intervention. We also show how the tf*idf weight vector can be used to both visualize the most interesting patterns and to highlight the parts of a given surgery that are the most interesting. CONCLUSIONS: Identifying patterns that discriminate groups of surgeon is a very important step in improving the understanding of surgical processes. The proposed method finds discriminative and interpretable patterns in sequences of surgical activities. Our approach provides intuitive results, as it identifies automatically the set of patterns explaining the differences between the groups.

Image analysis of immune cell patterns in the human mammary gland during the menstrual cycle refines lymphocytic lobulitis.

PURPOSE: To improve microscopic evaluation of immune cells relevant in breast cancer oncoimmunology, we aim at distinguishing normal infiltration patterns from lymphocytic lobulitis by advanced image analysis. We consider potential immune cell variations due to the menstrual cycle and oral contraceptives in non-neoplastic mammary gland tissue. METHODS: Lymphocyte and macrophage distributions were analyzed in the anatomical context of the resting mammary gland in immunohistochemically stained digital whole slide images obtained from 53 reduction mammoplasty specimens. Our image analysis workflow included automated regions of interest detection, immune cell recognition, and co-registration of regions of interest. RESULTS: In normal lobular epithelium, seven CD8[Formula: see text] lymphocytes per 100 epithelial cells were present on average and about 70% of this T-lymphocyte population was lined up along the basal cell layer in close proximity to the epithelium. The density of CD8[Formula: see text] T-cell was 1.6 fold higher in the luteal than in the follicular phase in spontaneous menstrual cycles and 1.4 fold increased under the influence of oral contraceptives, and not co-localized with epithelial proliferation. CD4[Formula: see text] T-cells were infrequent. Abundant CD163[Formula: see text] macrophages were widely spread, including the interstitial compartment, with minor variation during the menstrual cycle. CONCLUSIONS: Spatial patterns of different immune cell subtypes determine the range of normal, as opposed to inflammatory conditions of the breast tissue microenvironment. Advanced image analysis enables quantification of hormonal effects, refines lymphocytic lobulitis, and shows potential for comprehensive biopsy evaluation in oncoimmunology.

Automatic matching of surgeries to predict surgeons' next actions.

OBJECTIVE: More than half a million surgeries are performed every day worldwide, which makes surgery one of the most important component of global health care. In this context, the objective of this paper is to introduce a new method for the prediction of the possible next task that a surgeon is going to perform during surgery. MATERIAL AND METHOD: We formulate the problem as finding the optimal registration of a partial sequence to a complete reference sequence of surgical activities. We propose an efficient algorithm to find the optimal partial alignment and a prediction system using maximum a posteriori probability estimation and filtering. We also introduce a weighting scheme allowing to improve the predictions by taking into account the relative similarity between the current surgery and a set of pre-recorded surgeries. RESULTS: Our method is evaluated on two types of neurosurgical procedures: lumbar disc herniation removal and anterior cervical discectomy. Results show that our method outperformed the state of the art by predicting the next task that the surgeon will perform with 95% accuracy. CONCLUSIONS: This work shows that, even from the low-level description of surgeries and without other sources of information, it is often possible to predict the next surgical task when the conditions are consistent with the previously recorded surgeries. We also showed that our method is able to assess when there is actually a large divergence between the predictions and decide that it is not reasonable to make a prediction.

Distinguishing surgical behavior by sequential pattern discovery.

OBJECTIVE: Each surgical procedure is unique due to patient's and also surgeon's particularities. In this study, we propose a new approach to distinguish surgical behaviors between surgical sites, levels of expertise and individual surgeons thanks to a pattern discovery method. METHODS: The developed approach aims to distinguish surgical behaviors based on shared longest frequent sequential patterns between surgical process models. To allow clustering, we propose a new metric called SLFSP. The approach is validated by comparison with a clustering method using Dynamic Time Warping as a metric to characterize the similarity between surgical process models. RESULTS: Our method outperformed the existing approach. It was able to make a perfect distinction between surgical sites (accuracy of 100%). We reached an accuracy superior to 90% and 85% for distinguishing levels of expertise and individual surgeons. CONCLUSION: Clustering based on shared longest frequent sequential patterns outperformed the previous study based on time analysis. SIGNIFICANCE: The proposed method shows the feasibility of comparing surgical process models, not only by their duration but also by their structure of activities. Furthermore, patterns may show risky behaviors, which could be an interesting information for surgical training to prevent adverse events.

Detection of lobular structures in normal breast tissue.

BACKGROUND: Ongoing research into inflammatory conditions raises an increasing need to evaluate immune cells in histological sections in biologically relevant regions of interest (ROIs). Herein, we compare different approaches to automatically detect lobular structures in human normal breast tissue in digitized whole slide images (WSIs). This automation is required to perform objective and consistent quantitative studies on large data sets. METHODS: In normal breast tissue from nine healthy patients immunohistochemically stained for different markers, we evaluated and compared three different image analysis methods to automatically detect lobular structures in WSIs: (1) a bottom-up approach using the cell-based data for subsequent tissue level classification, (2) a top-down method starting with texture classification at tissue level analysis of cell densities in specific ROIs, and (3) a direct texture classification using deep learning technology. RESULTS: All three methods result in comparable overall quality allowing automated detection of lobular structures with minor advantage in sensitivity (approach 3), specificity (approach 2), or processing time (approach 1). Combining the outputs of the approaches further improved the precision. CONCLUSIONS: Different approaches of automated ROI detection are feasible and should be selected according to the individual needs of biomarker research. Additionally, detected ROIs could be used as a basis for quantification of immune infiltration in lobular structures.

Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training.

Dexterity and procedural knowledge are two critical skills that surgeons need to master to perform accurate and safe surgical interventions. However, current training systems do not allow us to provide an in-depth analysis of surgical gestures to precisely assess these skills. Our objective is to develop a method for the automatic and quantitative assessment of surgical gestures. To reach this goal, we propose a new unsupervised algorithm that can automatically segment kinematic data from robotic training sessions. Without relying on any prior information or model, this algorithm detects critical points in the kinematic data that define relevant spatio-temporal segments. Based on the association of these segments, we obtain an accurate recognition of the gestures involved in the surgical training task. We, then, perform an advanced analysis and assess our algorithm using datasets recorded during real expert training sessions. After comparing our approach with the manual annotations of the surgical gestures, we observe 97.4% accuracy for the learning purpose and an average matching score of 81.9% for the fully automated gesture recognition process. Our results show that trainees workflow can be followed and surgical gestures may be automatically evaluated according to an expert database. This approach tends toward improving training efficiency by minimizing the learning curve.

Automatic phase prediction from low-level surgical activities.

PURPOSE: Analyzing surgical activities has received a growing interest in recent years. Several methods have been proposed to identify surgical activities and surgical phases from data acquired in operating rooms. These context-aware systems have multiple applications including: supporting the surgical team during the intervention, improving the automatic monitoring, designing new teaching paradigms. METHODS: In this paper, we use low-level recordings of the activities that are performed by a surgeon to automatically predict the current (high-level) phase of the surgery. We augment a decision tree algorithm with the ability to consider the local context of the surgical activities and a hierarchical clustering algorithm. RESULTS: Experiments were performed on 22 surgeries of lumbar disk herniation. We obtained an overall precision of 0.843 in detecting phases of 51,489 single activities. We also assess the robustness of the method with regard to noise. CONCLUSION: We show that using the local context allows us to improve the results compared with methods only considering single activity. Experiments show that the use of the local context makes our method very robust to noise and that clustering the input data first improves the predictions.

Non-linear temporal scaling of surgical processes.

OBJECTIVE: Surgery is one of the riskiest and most important medical acts that is performed today. Understanding the ways in which surgeries are similar or different from each other is of major interest. Desires to improve patient outcomes and surgeon training, and to reduce the costs of surgery, all motivate a better understanding of surgical practices. To facilitate this, surgeons have started recording the activities that are performed during surgery. New methods have to be developed to be able to make the most of this extremely rich and complex data. The objective of this work is to enable the simultaneous comparison of a set of surgeries, in order to be able to extract high-level information about surgical practices. MATERIALS AND METHOD: We introduce non-linear temporal scaling (NLTS): a method that finds a multiple alignment of a set of surgeries. Experiments are carried out on a set of lumbar disc neurosurgeries. We assess our method both on a highly standardised phase of the surgery (closure) and on the whole surgery. RESULTS: Experiments show that NLTS makes it possible to consistently derive standards of surgical practice and to understand differences between groups of surgeries. We take the training of surgeons as the common theme for the evaluation of the results and highlight, for example, the main differences between the practices of junior and senior surgeons in the removal of a lumbar disc herniation. CONCLUSIONS: NLTS is an effective and efficient method to find a multiple alignment of a set of surgeries. NLTS realigns a set of sequences along their intrinsic timeline, which makes it possible to extract standards of surgical practices.