Effect of image resolution on automated classification of chest X-rays.

Purpose: Deep learning (DL) models have received much attention lately for their ability to achieve expert-level performance on the accurate automated analysis of chest X-rays (CXRs). Recently available public CXR datasets include high resolution images, but state-of-the-art models are trained on reduced size images due to limitations on graphics processing unit memory and training time. As computing hardware continues to advance, it has become feasible to train deep convolutional neural networks on high-resolution images without sacrificing detail by downscaling. This study examines the effect of increased resolution on CXR classification performance. Approach: We used the publicly available MIMIC-CXR-JPG dataset, comprising 377,110 high resolution CXR images for this study. We applied image downscaling from native resolution to 2048×2048 pixels, 1024×1024 pixels, 512×512 pixels, and 256×256 pixels and then we used the DenseNet121 and EfficientNet-B4 DL models to evaluate clinical task performance using these four downscaled image resolutions. Results: We find that while some clinical findings are more reliably labeled using high resolutions, many other findings are actually labeled better using downscaled inputs. We qualitatively verify that tasks requiring a large receptive field are better suited to downscaled low resolution input images, by inspecting effective receptive fields and class activation maps of trained models. Finally, we show that stacking an ensemble across resolutions outperforms each individual learner at all input resolutions while providing interpretable scale weights, indicating that diverse information is extracted across resolutions. Conclusions: This study suggests that instead of focusing solely on the finest image resolutions, multi-scale features should be emphasized for information extraction from high-resolution CXRs.

Deep Learning on Multimodal Chemical and Whole Slide Imaging Data for Predicting Prostate Cancer Directly from Tissue Images.

Prostate cancer is one of the most common cancers globally and is the second most common cancer in the male population in the US. Here we develop a study based on correlating the hematoxylin and eosin (H&E)-stained biopsy data with MALDI mass-spectrometric imaging data of the corresponding tissue to determine the cancerous regions and their unique chemical signatures and variations of the predicted regions with original pathological annotations. We obtain features from high-resolution optical micrographs of whole slide H&E stained data through deep learning and spatially register them with mass spectrometry imaging (MSI) data to correlate the chemical signature with the tissue anatomy of the data. We then use the learned correlation to predict prostate cancer from observed H&E images using trained coregistered MSI data. This multimodal approach can predict cancerous regions with ∼80% accuracy, which indicates a correlation between optical H&E features and chemical information found in MSI. We show that such paired multimodal data can be used for training feature extraction networks on H&E data which bypasses the need to acquire expensive MSI data and eliminates the need for manual annotation saving valuable time. Two chemical biomarkers were also found to be predicting the ground truth cancerous regions. This study shows promise in generating improved patient treatment trajectories by predicting prostate cancer directly from readily available H&E-stained biopsy images aided by coregistered MSI data.

Hypothesis-Agnostic Network-Based Analysis of Real-World Data Suggests Ondansetron is Associated with Lower COVID-19 Any Cause Mortality.

BACKGROUND: The COVID-19 pandemic generated a massive amount of clinical data, which potentially hold yet undiscovered answers related to COVID-19 morbidity, mortality, long-term effects, and therapeutic solutions. OBJECTIVES: The objectives of this study were (1) to identify novel predictors of COVID-19 any cause mortality by employing artificial intelligence analytics on real-world data through a hypothesis-agnostic approach and (2) to determine if these effects are maintained after adjusting for potential confounders and to what degree they are moderated by other variables. METHODS: A Bayesian statistics-based artificial intelligence data analytics tool (bAIcis®) within the Interrogative Biology® platform was used for Bayesian network learning and hypothesis generation to analyze 16,277 PCR+ patients from a database of 279,281 inpatients and outpatients tested for SARS-CoV-2 infection by antigen, antibody, or PCR methods during the first pandemic year in Central Florida. This approach generated Bayesian networks that enabled unbiased identification of significant predictors of any cause mortality for specific COVID-19 patient populations. These findings were further analyzed by logistic regression, regression by least absolute shrinkage and selection operator, and bootstrapping. RESULTS: We found that in the COVID-19 PCR+ patient cohort, early use of the antiemetic agent ondansetron was associated with decreased any cause mortality 30 days post-PCR+ testing in mechanically ventilated patients. CONCLUSIONS: The results demonstrate how a real-world COVID-19-focused data analysis using artificial intelligence can generate unexpected yet valid insights that could possibly support clinical decision making and minimize the future loss of lives and resources.

Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy.

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with domain applications ranging from theory and materials prediction to high-throughput data analysis. In parallel, the recent successes in applying ML/AI methods for autonomous systems from robotics to self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiments (AE) in imaging. Here, we aim to analyze the major pathways toward AE in imaging methods with sequential image formation mechanisms, focusing on scanning probe microscopy (SPM) and (scanning) transmission electron microscopy ((S)TEM). We argue that automated experiments should necessarily be discussed in a broader context of the general domain knowledge that both informs the experiment and is increased as the result of the experiment. As such, this analysis should explore the human and ML/AI roles prior to and during the experiment and consider the latencies, biases, and prior knowledge of the decision-making process. Similarly, such discussion should include the limitations of the existing imaging systems, including intrinsic latencies, non-idealities, and drifts comprising both correctable and stochastic components. We further pose that the role of the AE in microscopy is not the exclusion of human operators (as is the case for autonomous driving), but rather automation of routine operations such as microscope tuning, etc., prior to the experiment, and conversion of low latency decision making processes on the time scale spanning from image acquisition to human-level high-order experiment planning. Overall, we argue that ML/AI can dramatically alter the (S)TEM and SPM fields; however, this process is likely to be highly nontrivial and initiated by combined human-ML workflows and will bring challenges both from the microscope and ML/AI sides. At the same time, these methods will enable opportunities and paradigms for scientific discovery and nanostructure fabrication.

Autonomous Experiments in Scanning Probe Microscopy and Spectroscopy: Choosing Where to Explore Polarization Dynamics in Ferroelectrics.

Polarization dynamics in ferroelectric materials are explored via automated experiment in piezoresponse force microscopy/spectroscopy (PFM/S). A Bayesian optimization (BO) framework for imaging is developed, and its performance for a variety of acquisition and pathfinding functions is explored using previously acquired data. The optimized algorithm is then deployed on an operational scanning probe microscope (SPM) for finding areas of large electromechanical response in a thin film of PbTiO3, with results showing that, with just 20% of the area sampled, most high-response clusters were captured. This approach can allow performing more complex spectroscopies in SPM that were previously not possible due to time constraints and sample stability. Improvements to the framework to enable the incorporation of more prior information and improve efficiency further are modeled and discussed.

Classifying cancer pathology reports with hierarchical self-attention networks.

We introduce a deep learning architecture, hierarchical self-attention networks (HiSANs), designed for classifying pathology reports and show how its unique architecture leads to a new state-of-the-art in accuracy, faster training, and clear interpretability. We evaluate performance on a corpus of 374,899 pathology reports obtained from the National Cancer Institute's (NCI) Surveillance, Epidemiology, and End Results (SEER) program. Each pathology report is associated with five clinical classification tasks - site, laterality, behavior, histology, and grade. We compare the performance of the HiSAN against other machine learning and deep learning approaches commonly used on medical text data - Naive Bayes, logistic regression, convolutional neural networks, and hierarchical attention networks (the previous state-of-the-art). We show that HiSANs are superior to other machine learning and deep learning text classifiers in both accuracy and macro F-score across all five classification tasks. Compared to the previous state-of-the-art, hierarchical attention networks, HiSANs not only are an order of magnitude faster to train, but also achieve about 1% better relative accuracy and 5% better relative macro F-score.

Deep Transfer Learning Across Cancer Registries for Information Extraction from Pathology Reports.

Automated text information extraction from cancer pathology reports is an active area of research to support national cancer surveillance. A well-known challenge is how to develop information extraction tools with robust performance across cancer registries. In this study we investigated whether transfer learning (TL) with a convolutional neural network (CNN) can facilitate cross-registry knowledge sharing. Specifically, we performed a series of experiments to determine whether a CNN trained with single-registry data is capable of transferring knowledge to another registry or whether developing a cross-registry knowledge database produces a more effective and generalizable model. Using data from two cancer registries and primary tumor site and topography as the information extraction task of interest, our study showed that TL results in 6.90% and 17.22% improvement of classification macro F-score over the baseline single-registry models. Detailed analysis illustrated that the observed improvement is evident in the low prevalence classes.

Biomass accessibility analysis using electron tomography.

BACKGROUND: Substrate accessibility to catalysts has been a dominant theme in theories of biomass deconstruction. However, current methods of quantifying accessibility do not elucidate mechanisms for increased accessibility due to changes in microstructure following pretreatment. RESULTS: We introduce methods for characterization of surface accessibility based on fine-scale microstructure of the plant cell wall as revealed by 3D electron tomography. These methods comprise a general framework, enabling analysis of image-based cell wall architecture using a flexible model of accessibility. We analyze corn stover cell walls, both native and after undergoing dilute acid pretreatment with and without a steam explosion process, as well as AFEX pretreatment. CONCLUSION: Image-based measures provide useful information about how much pretreatments are able to increase biomass surface accessibility to a wide range of catalyst sizes. We find a strong dependence on probe size when measuring surface accessibility, with a substantial decrease in biomass surface accessibility to probe sizes above 5-10 nm radius compared to smaller probes.

Investigating the feasibility of rapid MRI for image-guided motion management in lung cancer radiotherapy.

Cycle-to-cycle variations in respiratory motion can cause significant geometric and dosimetric errors in the administration of lung cancer radiation therapy. A common limitation of the current strategies for motion management is that they assume a constant, reproducible respiratory cycle. In this work, we investigate the feasibility of using rapid MRI for providing long-term imaging of the thorax in order to better capture cycle-to-cycle variations. Two nonsmall-cell lung cancer patients were imaged (free-breathing, no extrinsic contrast, and 1.5 T scanner). A balanced steady-state-free-precession (b-SSFP) sequence was used to acquire cine-2D and cine-3D (4D) images. In the case of Patient 1 (right midlobe lesion, ~40 mm diameter), tumor motion was well correlated with diaphragmatic motion. In the case of Patient 2, (left upper-lobe lesion, ~60 mm diameter), tumor motion was poorly correlated with diaphragmatic motion. Furthermore, the motion of the tumor centroid was poorly correlated with the motion of individual points on the tumor boundary, indicating significant rotation and/or deformation. These studies indicate that image quality and acquisition speed of cine-2D MRI were adequate for motion monitoring. However, significant improvements are required to achieve comparable speeds for truly 4D MRI. Despite several challenges, rapid MRI offers a feasible and attractive tool for noninvasive, long-term motion monitoring.

Four-dimensional tissue deformation reconstruction (4D TDR) validation using a real tissue phantom.

Calculation of four-dimensional (4D) dose distributions requires the remapping of dose calculated on each available binned phase of the 4D CT onto a reference phase for summation. Deformable image registration (DIR) is usually used for this task, but unfortunately almost always considers only endpoints rather than the whole motion path. A new algorithm, 4D tissue deformation reconstruction (4D TDR), that uses either CT projection data or all available 4D CT images to reconstruct 4D motion data, was developed. The purpose of this work is to verify the accuracy of the fit of this new algorithm using a realistic tissue phantom. A previously described fresh tissue phantom with implanted electromagnetic tracking (EMT) fiducials was used for this experiment. The phantom was animated using a sinusoidal and a real patient-breathing signal. Four-dimensional computer tomography (4D CT) and EMT tracking were performed. Deformation reconstruction was conducted using the 4D TDR and a modified 4D TDR which takes real tissue hysteresis (4D TDR(Hysteresis)) into account. Deformation estimation results were compared to the EMT and 4D CT coordinate measurements. To eliminate the possibility of the high contrast markers driving the 4D TDR, a comparison was made using the original 4D CT data and data in which the fiducials were electronically masked. For the sinusoidal animation, the average deviation of the 4D TDR compared to the manually determined coordinates from 4D CT data was 1.9 mm, albeit with as large as 4.5 mm deviation. The 4D TDR calculation traces matched 95% of the EMT trace within 2.8 mm. The motion hysteresis generated by real tissue is not properly projected other than at endpoints of motion. Sinusoidal animation resulted in 95% of EMT measured locations to be within less than 1.2 mm of the measured 4D CT motion path, enabling accurate motion characterization of the tissue hysteresis. The 4D TDR(Hysteresis) calculation traces accounted well for the hysteresis and matched 95% of the EMT trace within 1.6 mm. An irregular (in amplitude and frequency) recorded patient trace applied to the same tissue resulted in 95% of the EMT trace points within less than 4.5 mm when compared to both the 4D CT and 4D TDR(Hysteresis) motion paths. The average deviation of 4D TDR(Hysteresis) compared to 4D CT datasets was 0.9 mm under regular sinusoidal and 1.0 mm under irregular patient trace animation. The EMT trace data fit to the 4D TDR(Hysteresis) was within 1.6 mm for sinusoidal and 4.5 mm for patient trace animation. While various algorithms have been validated for end-to-end accuracy, one can only be fully confident in the performance of a predictive algorithm if one looks at data along the full motion path. The 4D TDR, calculating the whole motion path rather than only phase- or endpoints, allows us to fully characterize the accuracy of a predictive algorithm, minimizing assumptions. This algorithm went one step further by allowing for the inclusion of tissue hysteresis effects, a real-world effect that is neglected when endpoint-only validation is performed. Our results show that the 4D TDR(Hysteresis) correctly models the deformation at the endpoints and any intermediate points along the motion path.

A VECTOR MOMENTA FORMULATION OF DIFFEOMORPHISMS FOR IMPROVED GEODESIC REGRESSION AND ATLAS CONSTRUCTION.

This paper presents a novel approach for diffeomorphic image regression and atlas estimation that results in improved convergence and numerical stability. We use a vector momenta representation of a diffeomorphism's initial conditions instead of the standard scalar momentum that is typically used. The corresponding variational problem results in a closed-form update for template estimation in both the geodesic regression and atlas estimation problems. While we show that the theoretical optimal solution is equivalent to the scalar momenta case, the simplification of the optimization problem leads to more stable and efficient estimation in practice. We demonstrate the effectiveness of our method for atlas estimation and geodesic regression using synthetically generated shapes and 3D MRI brain scans.

A hierarchical geodesic model for diffeomorphic longitudinal shape analysis.

Hierarchical linear models (HLMs) are a standard approach for analyzing data where individuals are measured repeatedly over time. However, such models are only applicable to longitudinal studies of Euclidean data. In this paper, we propose a novel hierarchical geodesic model (HGM), which generalizes HLMs to the manifold setting. Our proposed model explains the longitudinal trends in shapes represented as elements of the group of diffeomorphisms. The individual level geodesics represent the trajectory of shape changes within individuals. The group level geodesic represents the average trajectory of shape changes for the population. We derive the solution of HGMs on diffeomorphisms to estimate individual level geodesics, the group geodesic, and the residual geodesics. We demonstrate the effectiveness of HGMs for longitudinal analysis of synthetically generated shapes and 3D MRI brain scans.

IDiff: irrotational diffeomorphisms for computational anatomy.

The study of diffeomorphism groups is fundamental to computational anatomy, and in particular to image registration. One of the most developed frameworks employs a Riemannian-geometric approach using right-invariant Sobolev metrics. To date, the computation of the Riemannian log and exponential maps on the diffeomorphism group have been defined implicitly via an infinite-dimensional optimization problem. In this paper we the employ Brenier's (1991) polar factorization to decompose a diffeomorphism h as h(chi) = S o psi(chi), where psi = deltarho is the gradient of a convex function p and S epsilon SDiff(R(d)) is a volume-preserving diffeomorphism. We show that all such mappings psi form a submanifold, which we term IDiff(R(d)), generated by irrotational flows from the identity. Using the natural metric, the manifold IDiff(R(d)) is flat. This allows us to calculate the Riemannian log map on this submanifold of diffeomorphisms in closed form, and develop extremely efficient metric-based image registration algorithms. This result has far-reaching implications in terms of the statistical analysis of anatomical variability within the framework of computational anatomy.

Tissue characterization using a phantom to validate four-dimensional tissue deformation.

PURPOSE: This project proposes using a real tissue phantom for 4D tissue deformation reconstruction (4DTDR) and 4D deformable image registration (DIR) validation, which allows for the complete verification of the motion path rather than limited end-point to end-point of motion. METHODS: Three electro-magnetic-tracking (EMT) fiducials were implanted into fresh porcine liver that was subsequently animated in a clinically realistic phantom. The animation was previously shown to be similar to organ motion, including hysteresis, when driven using a real patient's breathing pattern. For this experiment, 4DCTs and EMT traces were acquired when the phantom was animated using both sinusoidal and recorded patient-breathing traces. Fiducial were masked prior to 4DTDR for reconstruction. The original 4DCT data (with fiducials) were sampled into 20 CT phase sets and fiducials' coordinates were recorded, resulting in time-resolved fiducial motion paths. Measured values of fiducial location were compared to EMT measured traces and the result calculated by 4DTDR. RESULTS: For the sinusoidal breathing trace, 95% of EMT measured locations were within 1.2 mm of the measured 4DCT motion path, allowing for repeatable accurate motion characterization. The 4DTDR traces matched 95% of the EMT trace within 1.6 mm. Using the more irregular (in amplitude and frequency) patient trace, 95% of the EMT trace points fitted both 4DCT and 4DTDR motion path within 4.5 mm. The average match of the 4DTDR estimation of the tissue hysteresis over all CT phases was 0.9 mm using a sinusoidal signal for animation and 1.0 mm using the patient trace. CONCLUSIONS: The real tissue phantom is a tool which can be used to accurately characterize tissue deformation, helping to validate or evaluate a DIR or 4DTDR algorithm over a complete motion path. The phantom is capable of validating, evaluating, and quantifying tissue hysteresis, thereby allowing for full motion path validation.

4D CT image reconstruction with diffeomorphic motion model.

Four-dimensional (4D) respiratory correlated computed tomography (RCCT) has been widely used for studying organ motion. Most current RCCT imaging algorithms use binning techniques that are susceptible to artifacts and challenge the quantitative analysis of organ motion. In this paper, we develop an algorithm for analyzing organ motion which uses the raw, time-stamped imaging data to reconstruct images while simultaneously estimating deformation in the subject's anatomy. This results in reduction of artifacts and facilitates a reduction in dose to the patient during scanning while providing equivalent or better image quality as compared to RCCT. The framework also incorporates fundamental physical properties of organ motion, such as the conservation of local tissue volume. We demonstrate that this approach is accurate and robust against noise and irregular breathing patterns. We present results for a simulated cone beam CT phantom, as well as a detailed real porcine liver phantom study demonstrating accuracy and robustness of the algorithm. An example of applying this algorithm to real patient image data is also presented.

Dosimetric evaluation of a "virtual" image-guidance alternative to explicit 6 degree of freedom robotic couch correction.

PURPOSE: Clinical evaluation of a "virtual" methodology for providing 6 degrees of freedom (6DOF) patient set-up corrections and comparison to corrections facilitated by a 6DOF robotic couch. METHODS: A total of 55 weekly in-room image-guidance computed tomographic (CT) scans were acquired using a CT-on-rails for 11 pelvic and head and neck cancer patients treated at our facility. Fusion of the CT-of-the-day to the simulation CT allowed prototype virtual 6DOF correction software to calculate the translations, single couch yaw, and beam-specific gantry and collimator rotations necessary to effectively reproduce the same corrections as a 6DOF robotic couch. These corrections were then used to modify the original treatment plan beam geometry and this modified plan geometry was applied to the CT-of-the-day to evaluate the dosimetric effects of the virtual correction method. This virtual correction dosimetry was compared with calculated geometric and dosimetric results for an explicit 6DOF robotic couch correction methodology. RESULTS: A (2%, 2mm) gamma analysis comparing dose distributions created using the virtual corrections to those from explicit corrections showed that an average of 95.1% of all points had a gamma of 1 or less, with a standard deviation of 3.4%. For a total of 470 dosimetric metrics (ie, maximum and mean dose statistics for all relevant structures) compared for all 55 image-guidance sessions, the average dose difference for these metrics between the plans employing the virtual corrections and the explicit corrections was -0.12% with a standard deviation of 0.82%; 97.9% of all metrics were within 2%. CONCLUSIONS: Results showed that the virtual corrections yielded dosimetric distributions that were essentially equivalent to those obtained when 6DOF robotic corrections were used, and that always outperformed the most commonly employed clinical approach of 3 translations only. This suggests that for the patient datasets studied here, highly effective image-guidance corrections can be made without the use of a robotic couch.

4D MAP image reconstruction incorporating organ motion.

Four-dimensional respiratory correlated computed tomography (4D RCCT) has been widely used for studying organ motion. Most current algorithms use binning techniques which introduce artifacts that can seriously hamper quantitative motion analysis. In this paper, we develop an algorithm for tracking organ motion which uses raw time-stamped data and simultaneously reconstructs images and estimates deformations in anatomy. This results in a reduction of artifacts and an increase in signal-to-noise ratio (SNR). In the case of CT, the increased SNR enables a reduction in dose to the patient during scanning. This framework also facilitates the incorporation of fundamental physical properties of organ motion, such as the conservation of local tissue volume. We show in this paper that this approach is accurate and robust against noise and irregular breathing for tracking organ motion. A detailed phantom study is presented, demonstrating accuracy and robustness of the algorithm. An example of applying this algorithm to real patient image data is also presented, demonstrating the utility of the algorithm in reducing artifacts.