Deep Learning to Quantify Pulmonary Edema in Chest Radiographs.

PURPOSE: To develop a machine learning model to classify the severity grades of pulmonary edema on chest radiographs. MATERIALS AND METHODS: In this retrospective study, 369 071 chest radiographs and associated radiology reports from 64 581 patients (mean age, 51.71 years; 54.51% women) from the MIMIC-CXR chest radiograph dataset were included. This dataset was split into patients with and without congestive heart failure (CHF). Pulmonary edema severity labels from the associated radiology reports were extracted from patients with CHF as four different ordinal levels: 0, no edema; 1, vascular congestion; 2, interstitial edema; and 3, alveolar edema. Deep learning models were developed using two approaches: a semisupervised model using a variational autoencoder and a pretrained supervised learning model using a dense neural network. Receiver operating characteristic curve analysis was performed on both models. RESULTS: The area under the receiver operating characteristic curve (AUC) for differentiating alveolar edema from no edema was 0.99 for the semisupervised model and 0.87 for the pretrained models. Performance of the algorithm was inversely related to the difficulty in categorizing milder states of pulmonary edema (shown as AUCs for semisupervised model and pretrained model, respectively): 2 versus 0, 0.88 and 0.81; 1 versus 0, 0.79 and 0.66; 3 versus 1, 0.93 and 0.82; 2 versus 1, 0.69 and 0.73; and 3 versus 2, 0.88 and 0.63. CONCLUSION: Deep learning models were trained on a large chest radiograph dataset and could grade the severity of pulmonary edema on chest radiographs with high performance.Supplemental material is available for this article.See also the commentary by Auffermann in this issue.© RSNA, 2021.

Multimodal Representation Learning via Maximization of Local Mutual Information.

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.

Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment.

We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time. Our experimental results suggest that the joint image-text representation learning improves the performance of pulmonary edema assessment compared to a supervised model trained on images only. We also show the use of the text for explaining the image classification by the joint model. To the best of our knowledge, our approach is the first to leverage free-text radiology reports for improving the image model performance in this application. Our code is available at: https://github.com/RayRuizhiLiao/joint_chestxray.

Performance of unscented Kalman filter tractography in edema: Analysis of the two-tensor model.

Diffusion MRI tractography is increasingly used in pre-operative neurosurgical planning to visualize critical fiber tracts. However, a major challenge for conventional tractography, especially in patients with brain tumors, is tracing fiber tracts that are affected by vasogenic edema, which increases water content in the tissue and lowers diffusion anisotropy. One strategy for improving fiber tracking is to use a tractography method that is more sensitive than the traditional single-tensor streamline tractography. We performed experiments to assess the performance of two-tensor unscented Kalman filter (UKF) tractography in edema. UKF tractography fits a diffusion model to the data during fiber tracking, taking advantage of prior information from the previous step along the fiber. We studied UKF performance in a synthetic diffusion MRI digital phantom with simulated edema and in retrospective data from two neurosurgical patients with edema affecting the arcuate fasciculus and corticospinal tracts. We compared the performance of several tractography methods including traditional streamline, UKF single-tensor, and UKF two-tensor. To provide practical guidance on how the UKF method could be employed, we evaluated the impact of using various seed regions both inside and outside the edematous regions, as well as the impact of parameter settings on the tractography sensitivity. We quantified the sensitivity of different methods by measuring the percentage of the patient-specific fMRI activation that was reached by the tractography. We expected that diffusion anisotropy threshold parameters, as well as the inclusion of a free water model, would significantly influence the reconstruction of edematous WM fiber tracts, because edema increases water content in the tissue and lowers anisotropy. Contrary to our initial expectations, varying the fractional anisotropy threshold and including a free water model did not affect the UKF two-tensor tractography output appreciably in these two patient datasets. The most effective parameter for increasing tracking sensitivity was the generalized anisotropy (GA) threshold, which increased the length of tracked fibers when reduced to 0.075. In addition, the most effective seeding strategy was seeding in the whole brain or in a large region outside of the edema. Overall, the main contribution of this study is to provide insight into how UKF tractography can work, using a two-tensor model, to begin to address the challenge of fiber tract reconstruction in edematous regions near brain tumors.

Frequency Diffeomorphisms for Efficient Image Registration.

This paper presents an efficient algorithm for large deformation diffeomorphic metric mapping (LDDMM) with geodesic shooting for image registration. We introduce a novel finite dimensional Fourier representation of diffeomorphic deformations based on the key fact that the high frequency components of a diffeomorphism remain stationary throughout the integration process when computing the deformation associated with smooth velocity fields. We show that manipulating high dimensional diffeomorphisms can be carried out entirely in the bandlimited space by integrating the nonstationary low frequency components of the displacement field. This insight substantially reduces the computational cost of the registration problem. Experimental results show that our method is significantly faster than the state-of-the-art diffeomorphic image registration methods while producing equally accurate alignment. We demonstrate our algorithm in two different applications of image registration: neuroimaging and in-utero imaging.

Temporal Registration in In-Utero Volumetric MRI Time Series.

We present a robust method to correct for motion and deformations in in-utero volumetric MRI time series. Spatio-temporal analysis of dynamic MRI requires robust alignment across time in the presence of substantial and unpredictable motion. We make a Markov assumption on the nature of deformations to take advantage of the temporal structure in the image data. Forward message passing in the corresponding hidden Markov model (HMM) yields an estimation algorithm that only has to account for relatively small motion between consecutive frames. We demonstrate the utility of the temporal model by showing that its use improves the accuracy of the segmentation propagation through temporal registration. Our results suggest that the proposed model captures accurately the temporal dynamics of deformations in in-utero MRI time series.