RFMiD: Retinal Image Analysis for multi-Disease Detection challenge.

In the last decades, many publicly available large fundus image datasets have been collected for diabetic retinopathy, glaucoma, and age-related macular degeneration, and a few other frequent pathologies. These publicly available datasets were used to develop a computer-aided disease diagnosis system by training deep learning models to detect these frequent pathologies. One challenge limiting the adoption of a such system by the ophthalmologist is, computer-aided disease diagnosis system ignores sight-threatening rare pathologies such as central retinal artery occlusion or anterior ischemic optic neuropathy and others that ophthalmologists currently detect. Aiming to advance the state-of-the-art in automatic ocular disease classification of frequent diseases along with the rare pathologies, a grand challenge on "Retinal Image Analysis for multi-Disease Detection" was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI - 2021). This paper, reports the challenge organization, dataset, top-performing participants solutions, evaluation measures, and results based on a new "Retinal Fundus Multi-disease Image Dataset" (RFMiD). There were two principal sub-challenges: disease screening (i.e. presence versus absence of pathology - a binary classification problem) and disease/pathology classification (a 28-class multi-label classification problem). It received a positive response from the scientific community with 74 submissions by individuals/teams that effectively entered in this challenge. The top-performing methodologies utilized a blend of data-preprocessing, data augmentation, pre-trained model, and model ensembling. This multi-disease (frequent and rare pathologies) detection will enable the development of generalizable models for screening the retina, unlike the previous efforts that focused on the detection of specific diseases.

Real-time estimation of lung deformation from body surface using a general CoordConv CNN.

BACKGROUND AND OBJECTIVE: Estimating the three-dimensional (3D) deformation of the lung is important for accurate dose delivery in radiotherapy and precise surgical guidance in lung surgery navigation. Additional 4D-CT information is often required to eliminate the effect of individual variations and obtain a more accurate estimation of lung deformation. However, this results in increased radiation dose. Therefore, we propose a novel method that estimates lung tissue deformation from depth maps and two CT phases per patient. METHODS: The method models the 3D motion of each voxel as a linear displacement along a direction vector, with a variable amplitude and phase that depend on the voxel location. The direction vector and amplitude are derived from the registration of the CT images at the end-of-exhale (EOE) and the end-of-inhale (EOI) phases. The voxel phase is estimated by a neural network. Coordinate convolution (CoordConv) is used to fuse multimodal data and embed absolute position information. The network takes the front and side views as well as the previous phase views as inputs to enhance accuracy. RESULTS: We evaluate the proposed method on two datasets: DIR-Lab and 4D-Lung, and obtain average errors of 2.11 mm and 1.36 mm, respectively. The method achieves real-time performance of less than 7 ms per frame on a NVIDIA GeForce 2080Ti GPU. CONCLUSION: Compared with previous methods, our method achieves comparable or even better accuracy with less CT phases.

Exhaustive matching of 3D/2D coronary artery structure based on imperfect segmentations.

PURPOSE: The 3D/2D coronary artery registration technique has been developed for the guidance of the percutaneous coronary intervention. It introduces the absent 3D structural information by fusing the pre-operative computed tomography angiography (CTA) volume with the intra-operative X-ray coronary angiography (XCA) image. To conduct the registration, an accurate matching of the coronary artery structures extracted from the two imaging modalities is an essential step. METHODS: In this study, we propose an exhaustive matching algorithm to solve this problem. First, by recognizing the fake bifurcations in the XCA image caused by projection and concatenating the fractured centerline fragments, the original XCA topological structure is restored. Then, the vessel segments in the two imaging modalities are removed orderly, which generates all the potential structures to simulate the imperfect segmentation results. Finally, the CTA and XCA structures are compared pairwise, and the matching result is obtained by searching for the structure pair with the minimum similarity score. RESULTS: The experiments were conducted based on a clinical dataset collected from 46 patients and comprising of 240 CTA/XCA data pairs. And the results show that the proposed method is very effective, which achieves an accuracy of 0.960 for recognizing the fake bifurcations in the XCA image and an accuracy of 0.896 for matching the CTA/XCA vascular structures. CONCLUSION: The proposed exhaustive structure matching algorithm is simple and straightforward without any impractical assumption or time-consuming computations. With this method, the influence of the imperfect segmentations is eliminated and the accurate matching could be achieved efficiently. This lays a good foundation for the subsequent 3D/2D coronary artery registration task.

S3R: Shape and Semantics-Based Selective Regularization for Explainable Continual Segmentation Across Multiple Sites.

In clinical practice, it is desirable for medical image segmentation models to be able to continually learn on a sequential data stream from multiple sites, rather than a consolidated dataset, due to storage cost and privacy restrictions. However, when learning on a new site, existing methods struggle with a weak memorizability for previous sites with complex shape and semantic information, and a poor explainability for the memory consolidation process. In this work, we propose a novel Shape and Semantics-based Selective Regularization ( [Formula: see text]) method for explainable cross-site continual segmentation to maintain both shape and semantic knowledge of previously learned sites. Specifically, [Formula: see text] method adopts a selective regularization scheme to penalize changes of parameters with high Joint Shape and Semantics-based Importance (JSSI) weights, which are estimated based on the parameter sensitivity to shape properties and reliable semantics of the segmentation object. This helps to prevent the related shape and semantic knowledge from being forgotten. Moreover, we propose an Importance Activation Mapping (IAM) method for memory interpretation, which indicates the spatial support for important parameters to visualize the memorized content. We have extensively evaluated our method on prostate segmentation and optic cup and disc segmentation tasks. Our method outperforms other comparison methods in reducing model forgetting and increasing explainability. Our code is available at https://github.com/jingyzhang/S3R.

Design, development and evaluation of an ergonomically designed dual-use mechanism for robot-assisted cardiovascular intervention.

PURPOSE: Robot-assisted cardiovascular intervention has been recently developed, which enables interventionists to avoid x-ray radiation and improve their comfort. However, there are still some challenges in the robotic design, such as the inability of the interventionist to freely perform natural clinical techniques and the limited motion travel of the interventional tool. To overcome these challenges, this paper proposes an ergonomically designed dual-use mechanism for cardiovascular intervention (DMCI). METHODS: DMCI can work as an ergonomic interface or a compact slave robot with unlimited motion travel. Our kinematic analysis of DMCI includes motion decoupling and coupling. Motion decoupling decomposes the translation and rotation from the interventionist's natural clinical actions at the master side. Motion coupling can calculate the input pulses of motors according to the desired rotation and translation, thus composing the motion of the intervention tool at the slave side. RESULTS: Our kinematic analysis of DMCI has been experimentally verified, where the overall mean rotational errors are all less than 1° and translational errors are all less than 1 mm. We also evaluated the performance of the DMCI-based master-slave system, where the overall rotational and translational errors are 0.821 ± 0.753° and 0.608 ± 0.512 mm. Moreover, operators were found to be generally more efficient when using the DMCI-based interface compared to the conventional joystick. CONCLUSION: We have validated our kinematic analysis of DMCI. The master-slave teleoperation experiment demonstrated that operators can freely perform natural clinical techniques through the DMCI-based interface, and the slave robot can replicate the operators' manipulation at the master side well.

Integrated Treatment Planning in Percutaneous Microwave Ablation of Lung Tumors.

Microwave ablation (MWA) is a clinically widespread minimally invasive treatment method for lung tumors. Preoperative planning plays a vital role in MWA therapy. However, previous planning methods are far from satisfactory in clinical practice because they only one-sidedly consider the surgical path or energy parameters of an MWA surgery. In this paper, we propose a novel planning model with a computational model of thermal damage to integrally optimize both the surgical path and energy parameters. To ensure the model can be solved in a reasonable time, we elaborate a search space reducing strategy based on clinical constraints. Simulation and ex vivo experimental results were compared with an average mean absolute error of 0.82 K and an average root mean square error of 1.01 K. Our planning model was evaluated on clinical data, and the experimental results demonstrate the effectiveness of our model.

Spine Segmentation with Multi-view GCN and Boundary Constraint.

Multi-class segmentation of vertebrae and inter-vertebral discs (IVDs) is crucial for the diagnosis and treatment of spinal diseases. However, it is still a challenge due to similarities between neighboring vertebrae of a subject and differences among the IVDs from different subjects. In this paper, we propose a novel spine segmentation framework to achieve automatic segmentation of vertebrae and IVDs in MR images. The core component of the new framework is a Multi-View GCN (MVGCN), which utilizes multi-view features and graph convolutional network (GCN) to reason about the relations of vertebrae and IVDs. We additionally use a boundary constraint for better segmentation of the boundary between vertebrae and IVDs. We test our method on a public spine dataset of 172 MR volumetric images for the vertebrae and IVDs segmentation. The experimental results demonstrate the efficacy of our method. Code and models of our method will be available in the future.

An automatic framework for estimating the pose of the catheter distal section using a coarse-to-fine network.

BACKGROUND AND OBJECTIVE: During percutaneous coronary intervention procedures, generally only 2D X-ray images are provided. The consequent lack of depth perception makes it difficult for interventionists to visually estimate the pose of medical tools inside the vasculature, especially for novices. Although some automatic methods have been developed to aid interventionists, it is still a challenging task to obtain stable and accurate pose estimation. In this paper, we describe a learning-based framework for estimating the pose of the catheter distal section (CDS). The main innovation of this framework is the proposal of a coarse-to-fine fusion network (CFF-Net) which can achieve the shape and orientation estimation for the CDS. METHODS: By adopting a two-step fusion, CFF-Net progressively solves the shape and orientation ambiguities. The first step is the early fusion where the 2D projection image fuses with the shape prior before input, which makes the estimated result own a specific catheter distal shape. The second step is the late fusion where CFF-Net fuse feature maps and the orientation data from Electromagnetic (EM) sensors to confirm the overall orientation of the CDS. Finally, the estimated pose in the EM space will be obtained after we combine the estimated shape and orientation from CFF-Net with the position information from the EM sensor. RESULTS: The effectiveness of CFF-Net has been verified in a simulated environment where RMSE of CFF-Net is 0.706 ± 0.121 mm. This approach was further transferred from simulation to reality using the real-world data, where RMSE of CFF-Net is 1.121 ± 0.124 mm and RMSE of the whole proposed framework is 1.577 ± 0.144 mm. CONCLUSION: In simulated and real-world experiments, our proposed approach has been proven to achieve high accuracy while ensuring real-time processing for estimating the pose of the CDS.

CAR-Net: A Deep Learning-Based Deformation Model for 3D/2D Coronary Artery Registration.

Percutaneous coronary intervention is widely applied for the treatment of coronary artery disease under the guidance of X-ray coronary angiography (XCA) image. However, the projective nature of XCA causes the loss of 3D structural information, which hinders the intervention. This issue can be addressed by the deformable 3D/2D coronary artery registration technique, which fuses the pre-operative computed tomography angiography volume with the intra-operative XCA image. In this study, we propose a deep learning-based neural network for this task. The registration is conducted in a segment-by-segment manner. For each vessel segment pair, the centerlines that preserve topological information are decomposed into an origin tensor and a spherical coordinate shape tensor as network input through independent branches. Features of different modalities are fused and processed for predicting angular deflections, which is a special type of deformation field implying motion and length preservation constraints for vessel segments. The proposed method achieves an average error of 1.13 mm on the clinical dataset, which shows the potential to be applied in clinical practice.

Target localization during respiration motion based on LSTM: A pilot study on robotic puncture system.

BACKGROUND: In the needle biopsy, the respiratory motion causes the displacement of thoracic-abdominal soft tissues, which brings great difficulty to accurate localization. Based on internal target motion and external marker motion, the existing methods need to establish a correlation model or a prediction model to compensate the respiratory movement, which can hardly achieve required accuracy in clinic use due to the complexity of the internal tissue motion. METHODS: In order to improve the tracking accuracy and reduce the number of models, we propose a framework for target localization based on long short-term memory (LSTM) method. Combined with the correlation model and the prediction model by using LSTM, we adopted the principal component of time-series features of external surrogate signals to predict the trajectory of the internal tumour target. Additionally, based on the electromagnetic tracking system and Universal Robots 3 robotic arm, we applied the proposed approach to a prototype of robotic puncture system for real-time tumour tracking. RESULTS: To verify the proposed method, experiments on both public datasets and customized motion phantom for respiratory simulation were performed. In the public dataset study, an average mean absolute error, and an average root-mean-square error of predictive results of 0.44 and 0.58 mm were achieved, respectively. In the motion phantom study, an average root mean square of puncturing error resulted in 0.65 mm. CONCLUSION: The experimental results demonstrate the proposed method improves the accuracy of target localization during respiratory movement and appeals the potentials applying to clinical application.

Deformable Registration of Coronary Arteries with Topological Constraints for Image-Guided Vascular Interventions.

2D/3D registration of preoperative computed tomography angiography with intra-operative X-ray angiography improves image guidance in percutaneous coronary intervention. However, previous registration methods are inaccurate and time-consuming due to simple deformation and iterative optimization, respectively. In this paper, we propose a novel method for non-rigid registration of coronary arteries based on a point set registration network, which predicts the complex deformation field directly without iterative optimization. In order to maintain the structure of coronary arteries, we advance the classical point set registration network with a loss function containing global and local topological constraints. The method was evaluated on ten clinical data, and it achieved a median chamfer distance of 73.60 pixels with a run time of less than 1s on CPU. Experimental results demonstrate that the proposed method is highly accurate and efficient.

Generation of a local lung respiratory motion model using a weighted sparse algorithm and motion prior-based registration.

Respiration-introduced tumor location uncertainty is a challenge in lung percutaneous interventions, especially for the respiratory motion estimation of the tumor and surrounding vessel structures. In this work, a local motion modeling method is proposed based on whole-chest computed tomography (CT) and CT-fluoroscopy (CTF) scans. A weighted sparse statistical modeling (WSSM) method that can accurately capture location errors for each landmark point is proposed for lung motion prediction. By varying the sparse weight coefficients of the WSSM method, newly input motion information is approximately represented by a sparse linear combination of the respiratory motion repository and employed to serve as prior knowledge for the following registration process. We have also proposed an adaptive motion prior-based registration method to improve the motion prediction accuracy of the motion model in the region of interest (ROI). This registration method adopts a B-spline scheme to interactively weight the relative influence of the prior knowledge, model surface and image intensity information by locally controlling the deformation in the CTF image region. The proposed method has been evaluated on 15 image pairs between the end-expiratory (EE) and end-inspiratory (EI) phases and 31 four-dimensional CT (4DCT) datasets. The results reveal that the proposed WSSM method achieved a better motion prediction performance than other existing lung statistical motion modeling methods, and the motion prior-based registration method can generate more accurate local motion information in the ROI.

A statistical weighted sparse-based local lung motion modelling approach for model-driven lung biopsy.

PURPOSE: Lung biopsy is currently the most effective procedure for cancer diagnosis. However, respiration-induced location uncertainty presents a challenge in precise lung biopsy. To reduce the medical image requirements for motion modelling, in this study, local lung motion information in the region of interest (ROI) is extracted from whole chest computed tomography (CT) and CT-fluoroscopy scans to predict the motion of potentially cancerous tissue and important vessels during the model-driven lung biopsy process. METHODS: The motion prior of the ROI was generated via a sparse linear combination of a subset of motion information from a respiratory motion repository, and a weighted sparse-based statistical model was used to preserve the local respiratory motion details. We also employed a motion prior-based registration method to improve the motion estimation accuracy in the ROI and designed adaptive variable coefficients to interactively weigh the relative influence of the prior knowledge and image intensity information during the registration process. RESULTS: The proposed method was applied to ten test subjects for the estimation of the respiratory motion field. The quantitative analysis resulted in a mean target registration error of 1.5 (0.8) mm and an average symmetric surface distance of 1.4 (0.6) mm. CONCLUSIONS: The proposed method shows remarkable advantages over traditional methods in preserving local motion details and reducing the estimation error in the ROI. These results also provide a benchmark for lung respiratory motion modelling in the literature.

Automatic detection of coronary artery stenosis by convolutional neural network with temporal constraint.

Coronary artery disease (CAD) is a major threat to human health. In clinical practice, X-ray coronary angiography remains the gold standard for CAD diagnosis, where the detection of stenosis is a crucial step. However, detection is challenging due to the low contrast between vessels and surrounding tissues as well as the complex overlap of background structures with inhomogeneous intensities. To achieve automatic and accurate stenosis detection, we propose a convolutional neural network-based method with a novel temporal constraint across X-ray angiographic sequences. Specifically, we develop a deconvolutional single-shot multibox detector for candidate detection on contrast-filled X-ray frames selected by U-Net. Based on these static frames, the detector demonstrates high sensitivity for stenoses yet unacceptable false positives still exist. To solve this problem, we propose a customized seq-fps module that exploits the temporal consistency of consecutive frames to reduce the number of false positives. Experiments are conducted with 148 X-ray angiographic sequences. The results show that the proposed method outperforms existing stenosis detection methods, achieving the highest sensitivity of 87.2% and positive predictive value of 79.5%. Furthermore, this study provides a promising tool to improve CAD diagnosis in clinical practice.

Prediction of late displacement of the globe in orbital blowout fractures.

PURPOSE: To establish a linear measuring method in computed tomographic (CT) images to predict the displacement of the globe late after orbital blowout fracture. METHODS: Subjects were retrospectively included. Inclusion criteria were as follows: (1) adult subjects (≥18 years old at the time of trauma); (2) unilateral orbital medial-wall and/or floor fractures; (3) CT examination at least 30 days after trauma. Exclusion criteria were as follows: (1) facial or orbital fracture extending to other parts of the orbit than medial-wall and/or floor; (2) history of orbital or ocular abnormality other than the orbital trauma; (3) severe ocular trauma accompanied by the orbital trauma; (4) orbital fracture treated surgically before the CT examination. A co-ordinate system was built based on the orbital CT scans. Displacements of orbital walls, displacement of the globe and relative location of the fracture site were measured. Correlations between the variables were investigated. RESULTS: Ninety-nine per cent of fracture sites of the medial wall and 100% of fracture sites of the floor were posterior to the centre of the unaffected globe. The affected globe moved significantly medially (p < 0.001) and backwards (p < 0.001) in pure medial-wall fracture; backwards (p < 0.001) and downwards (p = 0.017) in pure floor fracture; and medially (p < 0.001), backwards (p < 0.001) and downwards (p < 0.001) in medial-wall and floor fractures. Displacement of the globe was correlated with displacements of the orbital walls, and the regression formulae were therefore fitted. Application of the formulae revealed that the same extent of orbital wall displacement caused more displacement of the globe in female patients than in male patients. CONCLUSIONS: A linear measuring method in a three-dimensional co-ordinate system was established to identify the displacements of orbital walls and the displacement of the globe in orbital blowout fractures. The regression formulae generated in this study might be used in clinical practice to predict late displacement of the globe by measuring the displacements of orbital walls.

Spline curve deformation model with prior shapes for identifying adhesion boundaries between large lung tumors and tissues around lungs in CT images.

PURPOSE: Automated segmentation of lung tumors attached to anatomic structures such as the chest wall or mediastinum remains a technical challenge because of the similar Hounsfield units of these structures. To address this challenge, we propose herein a spline curve deformation model that combines prior shapes to correct large spatially contiguous errors (LSCEs) in input shapes derived from image-appearance cues.The model is then used to identify the adhesion boundaries between large lung tumors and tissue around the lungs. METHODS: The deformation of the whole curve is driven by the transformation of the control points (CPs) of the spline curve, which are influenced by external and internal forces. The external force drives the model to fit the positions of the non-LSCEs of the input shapes while the internal force ensures the local similarity of the displacements of the neighboring CPs. The proposed model corrects the gross errors in the lung input shape caused by large lung tumors, where the initial lung shape for the model is inferred from the training shapes by shape group-based sparse prior information and the input lung shape is inferred by adaptive-thresholding-based segmentation followed by morphological refinement. RESULTS: The accuracy of the proposed model is verified by applying it to images of lungs with either moderate large-sized (ML) tumors or giant large-sized (GL) tumors. The quantitative results in terms of the averages of the dice similarity coefficient (DSC) and the Jaccard similarity index (SI) are 0.982 ± 0.006 and 0.965 ± 0.012 for segmentation of lungs adhered by ML tumors, and 0.952 ± 0.048 and 0.926 ± 0.059 for segmentation of lungs adhered by GL tumors, which give 0.943 ± 0.021 and 0.897 ± 0.041 for segmentation of the ML tumors, and 0.907 ± 0.057 and 0.888 ± 0.091 for segmentation of the GL tumors, respectively. In addition, the bidirectional Hausdorff distances are 5.7 ± 1.4 and 11.3 ± 2.5 mm for segmentation of lungs with ML and GL tumors, respectively. CONCLUSIONS: When combined with prior shapes, the proposed spline curve deformation can deal with large spatially consecutive errors in object shapes obtained from image-appearance information. We verified this method by applying it to the segmentation of lungs with large tumors adhered to the tissue around the lungs and the large tumors. Both the qualitative and quantitative results are more accurate and repeatable than results obtained with current state-of-the-art techniques.

A novel robotic system for vascular intervention: principles, performances, and applications.

PURPOSE: This paper describes the design, principles, performances, and applications of a novel image-guided master-slave robotic system for vascular intervention (VI), including the performance evaluation and in vivo trials. METHODS: Based on the peer-to-peer (P2P) remote communication system, the kinetics analysis, the sliding-mode neural network self-adaptive control model and the feedback system, this new robotic system can accomplish in real time a number of VI operations, including guidewire translation and rotation, balloon catheter translation, and contrast agent injection. The master-slave design prevents surgeons from being exposed to X-ray radiation, which means that they are not required to wear a heavy lead suit. We also conducted a performance evaluation of the new system, which assessed the speed, position tracking, and accuracy, as well as in vivo swine trials. RESULTS: The speed and position tracking effects are really good, which contribute to the high level of performance in terms of the translational (error ≤ 0.45%) and rotational (error ≤ 2.6°) accuracy. In addition, the accuracy of the contrast agent injection is less than 0.2 ml. The robotic system successfully performed both the stent revascularization of an arteria carotis and four in vivo trials. The haptic feedback data correspond with the robotic-assisted procedure, and peaks and troughs of data occur regularly. CONCLUSIONS: By means of the performance evaluation and four successful in vivo trials, the feasibility and efficiency of the new robotic system are validated, which should prove helpful for further research.

Multi-sequence myocardium segmentation with cross-constrained shape and neural network-based initialization.

For myocardial infarction (MI) patients, delayed enhancement (DE) and T2-weighted cardiovascular magnetic resonance imaging (CMR) can play significant roles in diagnosis, prognosis and therapeutic strategy evaluation. However, the non-rigid registration between different CMR sequences is particularly challenging and prevents the use of multi-sequence image analysis. In this article, we propose an approach for segmenting T2 and DE CMR simultaneously with cross-constrained shape and shape discrepancy compensation. A framework for the unified segmentation of multi-sequence images is built based on the coupled level set method. Additionally, a sparse representation-based shape model is optimized under the constraints from both sequences for complementary information sharing. Considering the myocardium shape discrepancy between the two sequences due to non-perfect registration, an error term is added to explicitly model this difference. The intensity feature is extracted with a Gaussian mixture model from each sequence. To obtain a fully automatic approach, the conditional generative adversarial network is adopted for initialization. The results are evaluated with T2 and DE images from 32 MI patients. A promising Dice similarity coefficient of the myocardium is achieved (84.97±4.15% for T2 and 78.13±6.22% for DE CMR). This approach is a pilot work toward automatic, multi-sequence CMR image analysis.

A novel remote-controlled robotic system for cerebrovascular intervention.

BACKGROUND: In cerebrovascular intervention (CVI), the use of robots has considerable advantages over conventional surgery. This study introduces a remote-controlled robotic system, including the first in vivo proof-of-concept trial. METHODS: The robotic system uses a master-slave control strategy. Omega 3 was selected as the master manipulator, and the slave side executed the procedure of inserting the guidewire and balloon catheter, and angiography. The first in vivo trial was conducted to test whether the guidewire could be successfully moved from a pig's femoral artery to its carotid artery using our robotic system. RESULTS: The insertion of the guidewire and balloon catheter and the angiography were successfully accomplished without any vascular rupture. The guidewire was successfully inserted into the secondary branches of the pig's carotid. The robot-assisted surgery took a little more time than manual surgery. CONCLUSIONS: The successful first in vivo trial indicates the feasibility and effectiveness of the robotic system.