Magnetic resonance imaging-based deep learning imaging biomarker for predicting functional outcomes after acute ischemic stroke.

PURPOSE: Clinical risk scores are essential for predicting outcomes in stroke patients. The advancements in deep learning (DL) techniques provide opportunities to develop prediction applications using magnetic resonance (MR) images. We aimed to develop an MR-based DL imaging biomarker for predicting outcomes in acute ischemic stroke (AIS) and evaluate its additional benefit to current risk scores. METHOD: This study included 3338 AIS patients. We trained a DL model using deep neural network architectures on MR images and radiomics to predict poor functional outcomes at three months post-stroke. The DL model generated a DL score, which served as the DL imaging biomarker. We compared the predictive performance of this biomarker to five risk scores on a holdout test set. Additionally, we assessed whether incorporating the imaging biomarker into the risk scores improved the predictive performance. RESULTS: The DL imaging biomarker achieved an area under the receiver operating characteristic curve (AUC) of 0.788. The AUCs of the five studied risk scores were 0.789, 0.793, 0.804, 0.810, and 0.826, respectively. The imaging biomarker's predictive performance was comparable to four of the risk scores but inferior to one (p = 0.038). Adding the imaging biomarker to the risk scores improved the AUCs (p-values) to 0.831 (0.003), 0.825 (0.001), 0.834 (0.003), 0.836 (0.003), and 0.839 (0.177), respectively. The net reclassification improvement and integrated discrimination improvement indices also showed significant improvements (all p < 0.001). CONCLUSIONS: Using DL techniques to create an MR-based imaging biomarker is feasible and enhances the predictive ability of current risk scores.

Multimodal Sensors Enabled Autonomous Soft Robotic System with Self-Adaptive Manipulation.

Human hands are amazingly skilled at recognizing and handling objects of different sizes and shapes. To date, soft robots rarely demonstrate autonomy equivalent to that of humans for fine perception and dexterous operation. Here, an intelligent soft robotic system with autonomous operation and multimodal perception ability is developed by integrating capacitive sensors with triboelectric sensor. With distributed multiple sensors, our robot system can not only sense and memorize multimodal information but also enable an adaptive grasping method for robotic positioning and grasp control, during which the multimodal sensory information can be captured sensitively and fused at feature level for crossmodally recognizing objects, leading to a highly enhanced recognition capability. The proposed system, combining the performance and physical intelligence of biological systems (i.e., self-adaptive behavior and multimodal perception), will greatly advance the integration of soft actuators and robotics in many fields.

InertialNet: Inertial Measurement Learning for Simultaneous Localization and Mapping.

SLAM (simultaneous localization and mapping) plays a crucial role in autonomous robot navigation. A challenging aspect of visual SLAM systems is determining the 3D camera orientation of the motion trajectory. In this paper, we introduce an end-to-end network structure, InertialNet, which establishes the correlation between the image sequence and the IMU signals. Our network model is built upon inertial measurement learning and is employed to predict the camera's general motion pose. By incorporating an optical flow substructure, InertialNet is independent of the appearance of training sets and can be adapted to new environments. It maintains stable predictions even in the presence of image blur, changes in illumination, and low-texture scenes. In our experiments, we evaluated InertialNet on the public EuRoC dataset and our dataset, demonstrating its feasibility with faster training convergence and fewer model parameters for inertial measurement prediction.

Trajectory Tracking of Variable Centroid Objects Based on Fusion of Vision and Force Perception.

Compared with traditional rigid objects' dynamic throwing and catching by the robot, the in-flight trajectory of nonrigid objects (incredibly variable centroid objects) throwing is more challenging to predict and track. This article proposes a variable centroid trajectory tracking network (VCTTN) with the fusion of vision and force information by introducing force data of throw processing to the vision neural network. The VCTTN-based model-free robot control system is developed to perform highly precise prediction and tracking with a part of the in-flight vision. The flight trajectories dataset of variable centroid objects generated by the robot arm is collected to train VCTTN. The experimental results show that trajectory prediction and tracking with the vision-force VCTTN is superior to the ones with the traditional vision perception and has an excellent tracking performance.

Hybrid Visual-Ranging Servoing for Positioning Based on Image and Measurement Features.

In this article, a hybrid visual-ranging servoing method is proposed to realize high-precision positioning tasks with a 6-degree of freedom (DOF) manipulator. This method utilizes the image and measurement features directly in the control loop. Without the need of complex image feature design and attitude estimation, this method realizes the 6-DOF control of a robot. A vital challenge in traditional vision-based systems is avoiding local minima and singularity problems. To tackle this issue, a full-rank interaction matrix hybrid visual servo (FRHVS) design criterion is proposed, which guarantees that the hybrid interaction matrix and its pseudoinverse matrix are both full rank. Moreover, the interaction matrix for these hybrid strategies, which combines image features with other sensors features, is derived in an analytical form. Experiments on a 6-DOF manipulator show that the proposed method is effective and has global asymptotic stability and high precision.

Data Augmentation in Defect Detection of Sanitary Ceramics in Small and Non-i.i.d Datasets.

In this study, a data-augmentation method is proposed to narrow the significant difference between the distribution of training and test sets when small sample sizes are concerned. Two major obstacles exist in the process of defect detection on sanitary ceramics. The first results from the high cost of sample collection, namely, the difficulty in obtaining a large number of training images required by deep-learning algorithms, which limits the application of existing algorithms in sanitary-ceramic defect detection. Second, due to the limitation of production processes, the collected defect images are often marked, thereby resulting in great differences in distribution compared with the images of test sets, which further affects the performance of detect-detection algorithms. The lack of training data and the differences in distribution between training and test sets lead to the fact that existing deep learning-based algorithms cannot be used directly in the defect detection of sanitary ceramics. The method proposed in this study, which is based on a generative adversarial network and the Gaussian mixture model, can effectively increase the number of training samples and reduce distribution differences between training and test sets, and the features of the generated images can be controlled to a certain extent. By applying this method, the accuracy is improved from approximately 75% to nearly 90% in almost all experiments on different classification networks.

Diagnosis of Polypoidal Choroidal Vasculopathy From Fluorescein Angiography Using Deep Learning.

PURPOSE: To differentiate polypoidal choroidal vasculopathy (PCV) from choroidal neovascularization (CNV) and to determine the extent of PCV from fluorescein angiography (FA) using attention-based deep learning networks. METHODS: We build two deep learning networks for diagnosis of PCV using FA, one for detection and one for segmentation. Attention-gated convolutional neural network (AG-CNN) differentiates PCV from other types of wet age-related macular degeneration. Gradient-weighted class activation map (Grad-CAM) is generated to highlight important regions in the image for making the prediction, which offers explainability of the network. Attention-gated recurrent neural network (AG-PCVNet) for spatiotemporal prediction is applied for segmentation of PCV. RESULTS: AG-CNN is validated with a dataset containing 167 FA sequences of PCV and 70 FA sequences of CNV. AG-CNN achieves a classification accuracy of 82.80% at image-level, and 86.21% at patient-level for PCV. Grad-CAM shows that regions contributing to decision-making have on average 21.91% agreement with pathological regions identified by experts. AG-PCVNet is validated with 56 PCV sequences from the EVEREST-I study and achieves a balanced accuracy of 81.132% and dice score of 0.54. CONCLUSIONS: The developed software provides a means of performing detection and segmentation of PCV on FA images for the first time. This study is a promising step in changing the diagnostic procedure of PCV and therefore improving the detection rate of PCV using FA alone. TRANSLATIONAL RELEVANCE: The developed deep learning system enables early diagnosis of PCV using FA to assist the physician in choosing the best treatment for optimal visual prognosis.

Zebrafish Larva Orientation and Smooth Aspiration Control for Microinjection.

OBJECTIVE: The zebrafish has been proven to be a significant model organism in various research fields. For investigating the in vivo properties of biologics within zebrafish with developed organs, an automated zebrafish larva organ injection system is crucially needed. However, current zebrafish larva manipulation techniques cannot accomplish this operation efficiently and continuously. METHODS: In this paper, we present a novel zebrafish larva manipulation technique with two key steps in the microinjection system: orienting and aspirating zebrafish larvae automatically. The orientation control is realized in a customized microfluidic chip, after which the larva moves tail-first until reaching the channel exit. Then a dynamic model of larva aspiration is established and an adaptive robust controller is designed. RESULTS: Experimental results demonstrate that high success rate can be reached and damage to larva body is reduced. CONCLUSION: The presented strategy is capable of orienting and aspirating zebrafish larvae smoothly and efficiently. SIGNIFICANCE: The proposed methods have the advantage of low cost, easy implementability and good stability.

Hyaluronic acid on the urokinase sustained release with a hydrogel system composed of poloxamer 407: HA/P407 hydrogel system for drug delivery.

Pleural empyema is an inflammatory condition characterized by accumulation of pus inside the pleural cavity, which is usually followed by bacterial pneumonia. During the disease process, the pro-inflammatory and pro-fibrotic cytokines in the purulent pleural effusion cause proliferation of fibroblasts and deposition of extracellular matrix, which lead to fibrin deposition and fibrothorax. Urokinase instillation therapy through a chest drainage tube is frequently used for fibrinolysis in patients with empyema. However, urokinase treatment requires multiple instillation (2-3 times per day, for 4-8 days) and easily flows out from the chest drainage tube due to its high water solubility. In this in vitro study, we developed a thermo-responsive hydrogel based on poloxamer 407 (P407) combined with hyaluronic acid (HA) for optimal loading and release of urokinase. Our results show that the addition of HA to poloxamer gels provides a significantly more compact microstructure, with smaller pore sizes (**p < 0.001). The differential scanning calorimetry (DSC) profile revealed no influence on the micellization intensity of poloxamer gel by HA. The 25% poloxamer-based gel was significantly superior to the 23% poloxamer-based gel, with slower gel erosion when comparing the 16th hour residual gel weight of both gels (*p < 0.05; **p < 0.001). The 25% poloxamer-HA gel also exhibited a superior urokinase release profile and longer release time. A Fourier-transform infrared spectroscopy (FT-IR) study of the P407/HA hydrogel showed no chemical interactions between P407 and HA in the hydrogel system. The thermoresponsive P407/HA hydrogel may have a promising potential in the loading and delivery of hydrophilic drugs. On top of that, in vitro toxicity test of this combination demonstrates a lower toxicity.

Temporally Coherent Illumination Normalization for Indocyanine Green Video Angiography.

Indocyanine green (ICG) angiography is an imaging method for doctors to observe choroidal abnormalities in human eyes. The ICG angiograms typically exhibit inhomogeneous illumination, which poses serious difficulties for the development of computer-aided diagnostic tools. In this paper, we propose a novel illumination normalization method to alleviate the inhomogeneous illumination in ICG video angiograms. In particular, we first align the viewpoint of the input ICG video angiogram using an image registration method. Then, we acquire temporal information using time-dependent intrinsic image and compute the corresponding illumination image. Finally, we correct inhomogeneous illumination from the illumination image by estimating contrast and luminosity distortion. We have conducted extensive evaluation using ICG video angiograms of 60 patients. Two video quality assessment methods are utilized to evaluate the performance of our proposed illumination normalization method. The results show that our proposed method can help improve the visual quality of ICG video angiogram. Visual evaluation by a human expert also confirms that our method yields better illumination normalization results.

Visual Servoed Three-Dimensional Rotation Control in Zebrafish Larva Heart Microinjection System.

OBJECTIVE: Zebrafish larva heart microinjection is a widely used technique in cardiac disease study. Compared with intensively researched rotation control of spherical or nearly spherical targets with clear structures, such as cells and embryos, 3-D rotation control of zebrafish larva demands new techniques due to its nontransparent structures and irregular outlines. METHODS: In this paper, we present a vision-servo system to automate the rotation process of zebrafish larva body. A switched control strategy is adopted to rotate zebrafish larva about the optical axis by using two micropipettes. Precisely rolling about larva body is performed, which involves a custom-designed rotational micromanipulator. A vision detection and online tracking algorithm is also developed to meet the requirement of visual servoing. With designed rotation control strategy, zebrafish larva heart can be adjusted to a desired orientation, which is often towards the injection pipette tip. RESULTS: Experimental results show that the designed system is capable of achieving high success rate of 94% about -axis rotation and 100% about -axis with 50 trails. The system also performs an average speed of 44 s/larva with a satisfied rotation accuracy of 0.5 in the horizontal plane and 2.5 about its roll axis. CONCLUSION: The proposed strategy is effective in flexibly manipulating larvae in 3-D. SIGNIFICANCE: The developed 3-D rotation control scheme is able to be applied to injection of various organs in zebrafish larva body for different experimental requirements.

Foot-controlled robotic-enabled endoscope holder for endoscopic sinus surgery: A cadaveric feasibility study.

OBJECTIVES/HYPOTHESIS: To evaluate the feasibility of a unique prototype foot-controlled robotic-enabled endoscope holder (FREE) in functional endoscopic sinus surgery. STUDY DESIGN: Cadaveric study. METHODS: Using human cadavers, we investigated the feasibility, advantages, and disadvantages of the robotic endoscope holder in performing endoscopic sinus surgery with two hands in five cadaver heads, mimicking a single nostril three-handed technique. RESULTS: The FREE robot is relatively easy to use. Setup was quick, taking less than 3 minutes from docking the robot at the head of the bed to visualizing the middle meatus. The unit is also relatively small, takes up little space, and currently has four degrees of freedom. The learning curve for using the foot control was short. The use of both hands was not hindered by the presence of the endoscope in the nasal cavity. The tremor filtration also aided in the smooth movement of the endoscope, with minimal collisions. CONCLUSION: The FREE endoscope holder in an ex-vivo cadaver test corroborated the feasibility of the robotic prototype, which allows for a two-handed approach to surgery equal to a single nostril three-handed technique without the holder that may reduce operating time. Further studies will be needed to evaluate its safety profile and use in other areas of endoscopic surgery. LEVEL OF EVIDENCE: NA. Laryngoscope, 126:566-569, 2016.

Automatic Segmentation of Polypoidal Choroidal Vasculopathy from Indocyanine Green Angiography Using Spatial and Temporal Patterns.

PURPOSE: To develop a computer-aided diagnostic tool for automated detection and quantification of polypoidal regions in indocyanine green angiography (ICGA) images. METHODS: The ICGA sequences of 59 polypoidal choroidal vasculopathy (PCV) treatment-naïve patients from five Asian countries (Hong Kong, Singapore, South Korea, Taiwan, and Thailand) were provided by the EVEREST study. The ground truth was provided by the reading center for the presence of polypoidal regions. The proposed detection algorithm used both temporal and spatial features to characterize the severity of polypoidal lesions in ICGA sequences. Leave-one-out cross validation was carried out so that each patient was used once as the validation sample. For each patient, a fixed detection threshold of 0.5 on the severity was applied to obtain sensitivity, specificity, and balanced accuracy with respect to the ground truth. RESULTS: Our system achieved an average accuracy of 0.9126 (sensitivity = 0.9125, specificity = 0.9127) for detection of polyps in the 59 ICGA sequences. Among the total of 222 features extracted from ICGA sequence, the spatial variances exhibited best discriminative power in distinguishing between polyp and nonpolyp regions. The results also indicated the importance of combining spatial and temporal features to further improve detection accuracy. CONCLUSIONS: The developed software provided a means of detecting and quantifying polypoidal regions in ICGA images for the first time. TRANSLATIONAL RELEVANCE: This preliminary study demonstrated a computer-aided diagnostic tool, which enables objective evaluation of PCV and its progression. Ophthalmologists can easily visualize the polypoidal regions and obtain quantitative information about polyps by using the proposed system.

Complete healing of tracheoesophageal fistula in a ventilator-dependent patient by conservative treatment.

Acquired nonmalignant tracheoesophageal fistula (TEF) is a rare clinical condition with multiple etiologies, although post-intubation injury is the most common cause. TEFs can be fatal if left untreated due to devastating pulmonary complications caused by tracheobronchial contamination and poor nutrition. Herein, we present a case of complete healing of a post-intubation TEF under conservative treatment in a ventilator-dependent patient, and review previous studies regarding the treatment of acquired nonmalignant TEFs.

Real-time automatic fiducial marker tracking in low contrast cine-MV images.

PURPOSE: To develop a real-time automatic method for tracking implanted radiographic markers in low-contrast cine-MV patient images used in image-guided radiation therapy (IGRT). METHODS: Intrafraction motion tracking using radiotherapy beam-line MV images have gained some attention recently in IGRT because no additional imaging dose is introduced. However, MV images have much lower contrast than kV images, therefore a robust and automatic algorithm for marker detection in MV images is a prerequisite. Previous marker detection methods are all based on template matching or its derivatives. Template matching needs to match object shape that changes significantly for different implantation and projection angle. While these methods require a large number of templates to cover various situations, they are often forced to use a smaller number of templates to reduce the computation load because their methods all require exhaustive search in the region of interest. The authors solve this problem by synergetic use of modern but well-tested computer vision and artificial intelligence techniques; specifically the authors detect implanted markers utilizing discriminant analysis for initialization and use mean-shift feature space analysis for sequential tracking. This novel approach avoids exhaustive search by exploiting the temporal correlation between consecutive frames and makes it possible to perform more sophisticated detection at the beginning to improve the accuracy, followed by ultrafast sequential tracking after the initialization. The method was evaluated and validated using 1149 cine-MV images from two prostate IGRT patients and compared with manual marker detection results from six researchers. The average of the manual detection results is considered as the ground truth for comparisons. RESULTS: The average root-mean-square errors of our real-time automatic tracking method from the ground truth are 1.9 and 2.1 pixels for the two patients (0.26 mm/pixel). The standard deviations of the results from the 6 researchers are 2.3 and 2.6 pixels. The proposed framework takes about 128 ms to detect four markers in the first MV images and about 23 ms to track these markers in each of the subsequent images. CONCLUSIONS: The unified framework for tracking of multiple markers presented here can achieve marker detection accuracy similar to manual detection even in low-contrast cine-MV images. It can cope with shape deformations of fiducial markers at different gantry angles. The fast processing speed reduces the image processing portion of the system latency, therefore can improve the performance of real-time motion compensation.

Retinal vascular tree reconstruction with anatomical realism.

Motivated by the goals of automatically extracting vessel segments and constructing retinal vascular trees with anatomical realism, this paper presents and analyses an algorithm that combines vessel segmentation and grouping of the extracted vessel segments. The proposed method aims to restore the topology of the vascular trees with anatomical realism for clinical studies and diagnosis of retinal vascular diseases, which manifest abnormalities in either venous and/or arterial vascular systems. Vessel segments are grouped using extended Kalman filter which takes into account continuities in curvature, width, and intensity changes at the bifurcation or crossover point. At a junction, the proposed method applies the minimum-cost matching algorithm to resolve the conflict in grouping due to error in tracing. The system was trained with 20 images from the DRIVE dataset, and tested using the remaining 20 images. The dataset contained a mixture of normal and pathological images. In addition, six pathological fluorescein angiogram sequences were also included in this study. The results were compared against the groundtruth images provided by a physician, achieving average success rates of 88.79% and 90.09%, respectively.

Role of preoperative PET-CT in assessing mediastinal and hilar lymph node status in early stage lung cancer.

BACKGROUND: Positron emission tomography combined with computed tomography (PET-CT) is important in the assessment and workup of lung cancer staging. However, inconsistencies between clinical image results obtained and pathologic findings of surgical specimens are still very common, particularly in patients with clinical early stage lung cancer. We sought to clarify the role of PET-CT in predicting mediastinal lymph node status preoperatively in clinical early stage lung cancer patients. METHODS: The cases were collected retrospectively from January 2008 to February 2009. All patients were good surgical candidates, and clinically early-stage during the pre-op evaluation, which included CT, PET scan, and cardiopulmonary tests. All patients underwent surgery, with complete pathological evaluation of mediastinal lymph node (LNs). The pathological status and PET Standardized uptake value (SUV)(max) of mediastinal LNs were collected to calculate the ROC curve, and to determine the best cut-off value of PET SUV(max). Other cofactors, including sex, tumor size, tumor SUV(max), histology type, and lobar distribution, were analyzed utilizing correlation study, Chi-square test, and t-test for significance. RESULTS: A total of 83 patients were enrolled into the study. The majority of the cases were in pathological early stage (Stage I: 67.5%, Stage II: 12%). The cut-off point of mediastinal LN SUV(max) was 1.6 calculated by receiver operating characteristic (ROC) curve (sensitivity: 40%, specificity: 88.7%, negative predictive rate: 95.1%). The hilar LN SUV(max) was found to have a poor correlation to the final pathologic status of hilar nodes with insignificant p value (0.487). Tumor SUV(max) and increased hilar LN uptake (SUV(max) >  2.0) were found to be significantly correlated with the pathologic status of mediastinal LNs. The false positive rates by PET-CT scan in N1 and N2 nodes were 70% and 78%, respectively, primarily due to inflammatory process (as anthracosis the leading cause). CONCLUSION: Integrated PET-CT is a useful tool for predicting the negativity of mediastinal LN status pre-operatively in clinically early stage (Stages I and II) lung cancer but may be relatively inaccurate in predicting hilar LN status and largely confounded by false positives caused by inflammatory process.

Automatic characterization of classic choroidal neovascularization by using AdaBoost for supervised learning.

PURPOSE: To provide a computer-aided visualization tool for accurate diagnosis and quantification of choroidal neovascularization (CNV) on the basis of fluorescence leakage characteristics. METHODS: All image frames of a fluorescein angiography (FA) sequence are first aligned and mapped to a global space. To automatically determine the severity of each pixel in the global space and hence the extent of CNV, the system matches the intensity variation of each set of spatially corresponding pixels across the sequence with the targeted leakage pattern, learned from a sampled population graded by a retina specialist. The learning strategy, known as the AdaBoost algorithm, has 12 classifiers for 12 features that summarize the variation in fluorescence intensity over time. Given a new sequence, the severity map image is generated using the contribution scores of the 12 classifiers. Initialized with points of low and high severity, regions of CNV are delineated using the random walk algorithm. RESULTS: A dataset of 33 FA sequences of classic CNV showed the average accuracy of CNV delineation to be 83.26%. In addition, the 30- to 60-second interval provided the most reliable information for differentiating CNV from the background. Using eight sequences of multiple visits of four patients for evaluation of the postphotodynamic therapy (PDT), the statistics derived from the segmented regions correlate closely with the clinical observed changes. CONCLUSIONS: The clinician can easily visualize the temporal characteristics of CNV fluorescence leakage using the severity map, which is a two-dimensional summary of a complete FA sequence. The computer-aided tool allows objective evaluation and computation of statistical data from the automatic delineation for surgical assessment.

Tumor-to-tumor metastasis: maxillary sinus adenoid cystic carcinoma metastasizing to double primary lung adenocarcinoma.

Tumor-to-tumor metastasis is a rare occurrence. The most frequent metastatic donor is lung cancer, and the most frequent recipient is renal cell carcinoma. Here we present a patient in whom lung cancer served as a metastatic recipient and maxillary sinus adenoid cystic carcinoma acted as a metastatic donor. This patient has double primary lung cancer, which increased the complexity of this case.

Optimal linear combination of facial regions for improving identification performance.

This paper presents a novel 3-D multiregion face recognition algorithm that consists of new geometric summation invariant features and an optimal linear feature fusion method. A summation invariant, which captures local characteristics of a facial surface, is extracted from multiple subregions of a 3-D range image as the discriminative features. Similarity scores between two range images are calculated from the selected subregions. A novel fusion method that is based on a linear discriminant analysis is developed to maximize the verification rate by a weighted combination of these similarity scores. Experiments on the Face Recognition Grand Challenge V2.0 dataset show that this new algorithm improves the recognition performance significantly in the presence of facial expressions.