FRSR: Framework for real-time scene reconstruction in robot-assisted minimally invasive surgery.

3D reconstruction of the intra-operative scenes provides precise position information which is the foundation of various safety related applications in robot-assisted surgery, such as augmented reality. Herein, a framework integrated into a known surgical system is proposed to enhance the safety of robotic surgery. In this paper, we present a scene reconstruction framework to restore the 3D information of the surgical site in real time. In particular, a lightweight encoder-decoder network is designed to perform disparity estimation, which is the key component of the scene reconstruction framework. The stereo endoscope of da Vinci Research Kit (dVRK) is adopted to explore the feasibility of the proposed approach, and it provides the possibility for the migration to other Robot Operating System (ROS) based robot platforms due to the strong independence on hardware. The framework is evaluated using three different scenarios, including a public dataset (3018 pairs of endoscopic images), the scene from the dVRK endoscope in our lab as well as a self-made clinical dataset captured from an oncology hospital. Experimental results show that the proposed framework can reconstruct 3D surgical scenes in real time (25 FPS), and achieve high accuracy (2.69 ± 1.48 mm in MAE, 5.47 ± 1.34 mm in RMSE and 0.41 ± 0.23 in SRE, respectively). It demonstrates that our framework can reconstruct intra-operative scenes with high reliability of both accuracy and speed, and the validation of clinical data also shows its potential in surgery. This work enhances the state of art in 3D intra-operative scene reconstruction based on medical robot platforms. The clinical dataset has been released to promote the development of scene reconstruction in the medical image community.

Deep learning-based overall survival prediction model in patients with rare cancer: a case study for primary central nervous system lymphoma.

PURPOSE: Primary central nervous system lymphoma (PCNSL) is a rare, aggressive form of extranodal non-Hodgkin lymphoma. To predict the overall survival (OS) in advance is of utmost importance as it has the potential to aid clinical decision-making. Though radiomics-based machine learning (ML) has demonstrated the promising performance in PCNSL, it demands large amounts of manual feature extraction efforts from magnetic resonance images beforehand. deep learning (DL) overcomes this limitation. METHODS: In this paper, we tailored the 3D ResNet to predict the OS of patients with PCNSL. To overcome the limitation of data sparsity, we introduced data augmentation and transfer learning, and we evaluated the results using r stratified k-fold cross-validation. To explain the results of our model, gradient-weighted class activation mapping was applied. RESULTS: We obtained the best performance (the standard error) on post-contrast T1-weighted (T1Gd)-area under curve [Formula: see text], accuracy [Formula: see text], precision [Formula: see text], recall [Formula: see text] and F1-score [Formula: see text], while compared with ML-based models on clinical data and radiomics data, respectively, further confirming the stability of our model. Also, we observed that PCNSL is a whole-brain disease and in the cases where the OS is less than 1 year, it is more difficult to distinguish the tumor boundary from the normal part of the brain, which is consistent with the clinical outcome. CONCLUSIONS: All these findings indicate that T1Gd can improve prognosis predictions of patients with PCNSL. To the best of our knowledge, this is the first time to use DL to explain model patterns in OS classification of patients with PCNSL. Future work would involve collecting more data of patients with PCNSL, or additional retrospective studies on different patient populations with rare diseases, to further promote the clinical role of our model.

Radiomics-Based Machine Learning Model for Predicting Overall and Progression-Free Survival in Rare Cancer: A Case Study for Primary CNS Lymphoma Patients.

Primary Central Nervous System Lymphoma (PCNSL) is an aggressive neoplasm with a poor prognosis. Although therapeutic progresses have significantly improved Overall Survival (OS), a number of patients do not respond to HD-MTX-based chemotherapy (15-25%) or experience relapse (25-50%) after an initial response. The reasons underlying this poor response to therapy are unknown. Thus, there is an urgent need to develop improved predictive models for PCNSL. In this study, we investigated whether radiomics features can improve outcome prediction in patients with PCNSL. A total of 80 patients diagnosed with PCNSL were enrolled. A patient sub-group, with complete Magnetic Resonance Imaging (MRI) series, were selected for the stratification analysis. Following radiomics feature extraction and selection, different Machine Learning (ML) models were tested for OS and Progression-free Survival (PFS) prediction. To assess the stability of the selected features, images from 23 patients scanned at three different time points were used to compute the Interclass Correlation Coefficient (ICC) and to evaluate the reproducibility of each feature for both original and normalized images. Features extracted from Z-score normalized images were significantly more stable than those extracted from non-normalized images with an improvement of about 38% on average (p-value < 10-12). The area under the ROC curve (AUC) showed that radiomics-based prediction overcame prediction based on current clinical prognostic factors with an improvement of 23% for OS and 50% for PFS, respectively. These results indicate that radiomics features extracted from normalized MR images can improve prognosis stratification of PCNSL patients and pave the way for further study on its potential role to drive treatment choice.

Classification of multiple sclerosis clinical profiles using machine learning and grey matter connectome.

Purpose: The main goal of this study is to investigate the discrimination power of Grey Matter (GM) thickness connectome data between Multiple Sclerosis (MS) clinical profiles using statistical and Machine Learning (ML) methods. Materials and Methods: A dataset composed of 90 MS patients acquired at the MS clinic of Lyon Neurological Hospital was used for the analysis. Four MS profiles were considered, corresponding to Clinical Isolated Syndrome (CIS), Relapsing-Remitting MS (RRMS), Secondary Progressive MS (SPMS), and Primary Progressive MS (PPMS). Each patient was classified in one of these profiles by our neurologist and underwent longitudinal MRI examinations including T1-weighted image acquisition at each examination, from which the GM tissue was segmented and the cortical GM thickness measured. Following the GM parcellation using two different atlases (FSAverage and Glasser 2016), the morphological connectome was built and six global metrics (Betweenness Centrality (BC), Assortativity (r), Transitivity (T), Efficiency (E g ), Modularity (Q) and Density (D)) were extracted. Based on their connectivity metrics, MS profiles were first statistically compared and second, classified using four different learning machines (Logistic Regression, Random Forest, Support Vector Machine and AdaBoost), combined in a higher level ensemble model by majority voting. Finally, the impact of the GM spatial resolution on the MS clinical profiles classification was analyzed. Results: Using binary comparisons between the four MS clinical profiles, statistical differences and classification performances higher than 0.7 were observed. Good performances were obtained when comparing the two early clinical forms, RRMS and PPMS (F1 score of 0.86), and the two neurodegenerative profiles, PPMS and SPMS (F1 score of 0.72). When comparing the two atlases, slightly better performances were obtained with the Glasser 2016 atlas, especially between RRMS with PPMS (F1 score of 0.83), compared to the FSAverage atlas (F1 score of 0.69). Also, the thresholding value for graph binarization was investigated suggesting more informative graph properties in the percentile range between 0.6 and 0.8. Conclusion: An automated pipeline was proposed for the classification of MS clinical profiles using six global graph metrics extracted from the GM morphological connectome of MS patients. This work demonstrated that GM morphological connectivity data could provide good classification performances by combining four simple ML models, without the cost of long and complex MR techniques, such as MR diffusion, and/or deep learning architectures.

Lesion segmentation in lung CT scans using unsupervised adversarial learning.

Lesion segmentation in medical images is difficult yet crucial for proper diagnosis and treatment. Identifying lesions in medical images is costly and time-consuming and requires highly specialized knowledge. For this reason, supervised and semi-supervised learning techniques have been developed. Nevertheless, the lack of annotated data, which is common in medical imaging, is an issue; in this context, interesting approaches can use unsupervised learning to accurately distinguish between healthy tissues and lesions, training the network without using the annotations. In this work, an unsupervised learning technique is proposed to automatically segment coronavirus disease 2019 (COVID-19) lesions on 2D axial CT lung slices. The proposed approach uses the technique of image translation to generate healthy lung images based on the infected lung image without the need for lesion annotations. Attention masks are used to improve the quality of the segmentation further. Experiments showed the capability of the proposed approaches to segment the lesions, and it outperforms a range of unsupervised lesion detection approaches. The average reported results for the test dataset based on the metrics: Dice Score, Sensitivity, Specificity, Structure Measure, Enhanced-Alignment Measure, and Mean Absolute Error are 0.695, 0.694, 0.961, 0.791, 0.875, and 0.082 respectively. The achieved results are promising compared with the state-of-the-art and could constitute a valuable tool for future developments.

Robot-assisted ex vivo neobladder reconstruction: preliminary results of surgical skill evaluation.

PURPOSE: Advanced developments in the medical field have gradually increased the public demand for surgical skill evaluation. However, this assessment always depends on the direct observation of experienced surgeons, which is time-consuming and variable. The introduction of robot-assisted surgery provides a new possibility for this evaluation paradigm. This paper aims at evaluating surgeon performance automatically with novel evaluation metrics based on different surgical data. METHODS: Urologists ([Formula: see text]) from a hospital were requested to perform a simplified neobladder reconstruction on an ex vivo setup twice with different camera modalities ([Formula: see text]) randomly. They were divided into novices and experts ([Formula: see text], respectively) according to their experience in robot-assisted surgeries. Different performance metrics ([Formula: see text]) are proposed to achieve the surgical skill evaluation, considering both instruments and endoscope. Also, nonparametric tests are adopted to check if there are significant differences when evaluating surgeons performance. RESULTS: When grouping according to four stages of neobladder reconstruction, statistically significant differences can be appreciated in phase 1 ([Formula: see text]) and phase 2 ([Formula: see text]) with normalized time-related metrics and camera movement-related metrics, respectively. On the other hand, considering experience grouping shows that both metrics are able to highlight statistically significant differences between novice and expert performances in the control protocol. It also shows that the camera-related performance of experts is significantly different ([Formula: see text]) when handling the endoscope manually and when it is automatic. CONCLUSION: Surgical skill evaluation, using the approach in this paper, can effectively measure surgical procedures of surgeons with different experience. Preliminary results demonstrate that different surgical data can be fully utilized to improve the reliability of surgical evaluation. It also demonstrates its versatility and potential in the quantitative assessment of various surgical operations.

Ensemble Learning for Multiple Sclerosis Disability Estimation Using Brain Structural Connectivity.

Background: Multiple sclerosis (MS) is an autoimmune inflammatory disease of the central nervous system characterized by demyelination and neurodegeneration processes. It leads to different clinical courses and degrees of disability that need to be anticipated by the neurologist for personalized therapy. Recently, machine learning (ML) techniques have reached a high level of performance in brain disease diagnosis and/or prognosis, but the decision process of a trained ML system is typically nontransparent. Using brain structural connectivity data, a fully automatic ensemble learning model, augmented with an interpretable model, is proposed for the estimation of MS patients' disability, measured by the Expanded Disability Status Scale (EDSS). Materials and Methods: An ensemble of four boosting-based models (GBM, XGBoost, CatBoost, and LightBoost) organized following a stacking generalization scheme was developed using diffusion tensor imaging (DTI)-based structural connectivity data. In addition, an interpretable model based on conditional logistic regression was developed to explain the best performances in terms of white matter (WM) links for three classes of EDSS (low, medium, and high). Results: The ensemble model reached excellent level of performance (root mean squared error of 0.92 ± 0.28) compared with single-based models and provided a better EDSS estimation using DTI-based structural connectivity data compared with conventional magnetic resonance imaging measures associated with patient data (age, gender, and disease duration). Used for interpretation of the estimation process, the counterfactual method showed the importance of certain brain networks, corresponding mainly to left hemisphere WM links, connecting the left superior temporal with the left posterior cingulate and the right precuneus gray matter regions, and the interhemispheric WM links constituting the corpus callosum. Also, a better accuracy estimation was found for the high disability class. Conclusion: The combination of advanced ML models and sensitive techniques such as DTI-based structural connectivity demonstrated to be useful for the estimation of MS patients' disability and to point out the most important brain WM networks involved in disability. Impact statement An ensemble of "boosting" machine learning (ML) models was more performant than single models to estimate disability in multiple sclerosis. Diffusion tensor imaging (DTI)-based structural connectivity led to better performance than conventional magnetic resonance imaging. An interpretable model, based on counterfactual perturbation, highlighted the most relevant white matter fiber links for disability estimation. These findings demonstrated the clinical interest of combining DTI, graph modeling, and ML techniques.

Data augmentation using generative adversarial neural networks on brain structural connectivity in multiple sclerosis.

BACKGROUND AND OBJECTIVE: Machine learning frameworks have demonstrated their potentials in dealing with complex data structures, achieving remarkable results in many areas, including brain imaging. However, a large collection of data is needed to train these models. This is particularly challenging in the biomedical domain since, due to acquisition accessibility, costs and pathology related variability, available datasets are limited and usually imbalanced. To overcome this challenge, generative models can be used to generate new data. METHODS: In this study, a framework based on generative adversarial network is proposed to create synthetic structural brain networks in Multiple Sclerosis (MS). The dataset consists of 29 relapsing-remitting and 19 secondary-progressive MS patients. T1 and diffusion tensor imaging (DTI) acquisitions were used to obtain the structural brain network for each subject. Evaluation of the quality of newly generated brain networks is performed by (i) analysing their structural properties and (ii) studying their impact on classification performance. RESULTS: We demonstrate that advanced generative models could be directly applied to the structural brain networks. We quantitatively and qualitatively show that newly generated data do not present significant differences compared to the real ones. In addition, augmenting the existing dataset with generated samples leads to an improvement of the classification performance (F1score 81%) with respect to the baseline approach (F1score 66%). CONCLUSIONS: Our approach defines a new tool for biomedical application when connectome-based data augmentation is needed, providing a valid alternative to usual image-based data augmentation techniques.

Using spatial-temporal ensembles of convolutional neural networks for lumen segmentation in ureteroscopy.

PURPOSE: Ureteroscopy is an efficient endoscopic minimally invasive technique for the diagnosis and treatment of upper tract urothelial carcinoma. During ureteroscopy, the automatic segmentation of the hollow lumen is of primary importance, since it indicates the path that the endoscope should follow. In order to obtain an accurate segmentation of the hollow lumen, this paper presents an automatic method based on convolutional neural networks (CNNs). METHODS: The proposed method is based on an ensemble of 4 parallel CNNs to simultaneously process single and multi-frame information. Of these, two architectures are taken as core-models, namely U-Net based in residual blocks ([Formula: see text]) and Mask-RCNN ([Formula: see text]), which are fed with single still-frames I(t). The other two models ([Formula: see text], [Formula: see text]) are modifications of the former ones consisting on the addition of a stage which makes use of 3D convolutions to process temporal information. [Formula: see text], [Formula: see text] are fed with triplets of frames ([Formula: see text], I(t), [Formula: see text]) to produce the segmentation for I(t). RESULTS: The proposed method was evaluated using a custom dataset of 11 videos (2673 frames) which were collected and manually annotated from 6 patients. We obtain a Dice similarity coefficient of 0.80, outperforming previous state-of-the-art methods. CONCLUSION: The obtained results show that spatial-temporal information can be effectively exploited by the ensemble model to improve hollow lumen segmentation in ureteroscopic images. The method is effective also in the presence of poor visibility, occasional bleeding, or specular reflections.

An Open-Source COVID-19 CT Dataset with Automatic Lung Tissue Classification for Radiomics.

The coronavirus disease 19 (COVID-19) pandemic is having a dramatic impact on society and healthcare systems. In this complex scenario, lung computerized tomography (CT) may play an important prognostic role. However, datasets released so far present limitations that hamper the development of tools for quantitative analysis. In this paper, we present an open-source lung CT dataset comprising information on 50 COVID-19-positive patients. The CT volumes are provided along with (i) an automatic threshold-based annotation obtained with a Gaussian mixture model (GMM) and (ii) a scoring provided by an expert radiologist. This score was found to significantly correlate with the presence of ground glass opacities and the consolidation found with GMM. The dataset is freely available in an ITK-based file format under the CC BY-NC 4.0 license. The code for GMM fitting is publicly available, as well. We believe that our dataset will provide a unique opportunity for researchers working in the field of medical image analysis, and hope that its release will lay the foundations for the successfully implementation of algorithms to support clinicians in facing the COVID-19 pandemic.

Microstructural Changes in the Left Mesocorticolimbic Pathway are Associated with the Comorbid Development of Fatigue and Depression in Multiple Sclerosis.

BACKGROUND AND PURPOSE: Lower reward responsiveness has been associated with fatigue in multiple sclerosis (MS). However, association of MS-related fatigue with damage to the mesocorticolimbic reward pathway (superolateral medial forebrain bundle [slMFB]) has not been assessed. We investigated the association of fatigue and depression with slMFB damage in MS patients stratified based on longitudinal fatigue patterns. METHODS: Patient stratification: 1. Sustained Fatigue (SF): latest two Modified Fatigue Impact Scale (MFIS) ≥ 38 (n = 26); 2. Reversible Fatigue (RF): latest MFIS < 38, and at least one previous MFIS ≥ 38 (n = 25); 3. Never Fatigued (NF): ≥ 5 consecutive MFIS < 38 (n = 42); 4. Healthy Controls (n = 6). Diffusion MRI-derived measures of fractional anisotropy (FA), axial (AD), mean (MD), and radial diffusivity (RD) of the slMFB were compared between the groups. Depression was assessed using the Center for Epidemiologic Studies-Depression Scale (CES-D). RESULTS: Depressed (CES-D ≥ 16) SF patients showed significantly higher MD and RD than nondepressed SF and RF, and depressed RF patients, and significantly lower FA than nondepressed SF and depressed RF patients in their left slMFB. Depressed SF patients showed significantly higher left slMFB MD and AD than healthy controls. CONCLUSION: Microstructural changes to the left slMFB may play a role in the comorbid development of fatigue and depression in MS.

Towards realistic laparoscopic image generation using image-domain translation.

Background and ObjectivesOver the last decade, Deep Learning (DL) has revolutionized data analysis in many areas, including medical imaging. However, there is a bottleneck in the advancement of DL in the surgery field, which can be seen in a shortage of large-scale data, which in turn may be attributed to the lack of a structured and standardized methodology for storing and analyzing surgical images in clinical centres. Furthermore, accurate annotations manually added are expensive and time consuming. A great help can come from the synthesis of artificial images; in this context, in the latest years, the use of Generative Adversarial Neural Networks (GANs) achieved promising results in obtaining photo-realistic images. MethodsIn this study, a method for Minimally Invasive Surgery (MIS) image synthesis is proposed. To this aim, the generative adversarial network pix2pix is trained to generate paired annotated MIS images by transforming rough segmentation of surgical instruments and tissues into realistic images. An additional regularization term was added to the original optimization problem, in order to enhance realism of surgical tools with respect to the background. Results Quantitative and qualitative (i.e., human-based) evaluations of generated images have been carried out in order to assess the effectiveness of the method. ConclusionsExperimental results show that the proposed method is actually able to translate MIS segmentations to realistic MIS images, which can in turn be used to augment existing data sets and help at overcoming the lack of useful images; this allows physicians and algorithms to take advantage from new annotated instances for their training.

Artificial intelligence for brain diseases: A systematic review.

Artificial intelligence (AI) is a major branch of computer science that is fruitfully used for analyzing complex medical data and extracting meaningful relationships in datasets, for several clinical aims. Specifically, in the brain care domain, several innovative approaches have achieved remarkable results and open new perspectives in terms of diagnosis, planning, and outcome prediction. In this work, we present an overview of different artificial intelligent techniques used in the brain care domain, along with a review of important clinical applications. A systematic and careful literature search in major databases such as Pubmed, Scopus, and Web of Science was carried out using "artificial intelligence" and "brain" as main keywords. Further references were integrated by cross-referencing from key articles. 155 studies out of 2696 were identified, which actually made use of AI algorithms for different purposes (diagnosis, surgical treatment, intra-operative assistance, and postoperative assessment). Artificial neural networks have risen to prominent positions among the most widely used analytical tools. Classic machine learning approaches such as support vector machine and random forest are still widely used. Task-specific algorithms are designed for solving specific problems. Brain images are one of the most used data types. AI has the possibility to improve clinicians' decision-making ability in neuroscience applications. However, major issues still need to be addressed for a better practical use of AI in the brain. To this aim, it is important to both gather comprehensive data and build explainable AI algorithms.

Classification of Multiple Sclerosis Clinical Profiles via Graph Convolutional Neural Networks.

Recent advances in image acquisition and processing techniques, along with the success of novel deep learning architectures, have given the opportunity to develop innovative algorithms capable to provide a better characterization of neurological related diseases. In this work, we introduce a neural network based approach to classify Multiple Sclerosis (MS) patients into four clinical profiles. Starting from their structural connectivity information, obtained by diffusion tensor imaging and represented as a graph, we evaluate the classification performances using unweighted and weighted connectivity matrices. Furthermore, we investigate the role of graph-based features for a better characterization and classification of the pathology. Ninety MS patients (12 clinically isolated syndrome, 30 relapsing-remitting, 28 secondary-progressive, and 20 primary-progressive) along with 24 healthy controls, were considered in this study. This work shows the great performances achieved by neural networks methods in the classification of the clinical profiles. Furthermore, it shows local graph metrics do not improve the classification results suggesting that the latent features created by the neural network in its layers have a much important informative content. Finally, we observe that graph weights representation of brain connections preserve important information to discriminate between clinical forms.

Prediction of Multiple Sclerosis Patient Disability from Structural Connectivity using Convolutional Neural Networks.

Prediction of disability progression in multiple sclerosis patients is a critical component of their management. In particular, one challenge is to identify and characterize a patient profile who may benefit of efficient treatments. However, it is not yet clear whether a particular relation exists between the brain structure and the disability status.This work aims at producing a fully automatic model for the expanded disability status score estimation, given the brain structural connectivity representation of a multiple sclerosis patient. The task is addressed by first extracting the connectivity graph, obtained by combining brain grey matter parcellation and tractography extracted from Diffusion and T1-weighted Magnetic Resonance (MR) images, and then processing it via a convolutional neural network (CNN) in order to compute the predicted score. Experiments show that the herein proposed approach achieves promising results, thus resulting as an important step forward on the road to better predict the evolution of the disease.