Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses.

Saliency maps are popularly used to "explain" decisions made by modern machine learning models, including deep convolutional neural networks (DCNNs). While the resulting heatmaps purportedly indicate important image features, their "trustworthiness," i.e., utility and robustness, has not been evaluated for musculoskeletal imaging. The purpose of this study was to systematically evaluate the trustworthiness of saliency maps used in disease diagnosis on upper extremity X-ray images. The underlying DCNNs were trained using the Stanford MURA dataset. We studied four trustworthiness criteria-(1) localization accuracy of abnormalities, (2) repeatability, (3) reproducibility, and (4) sensitivity to underlying DCNN weights-across six different gradient-based saliency methods (Grad-CAM (GCAM), gradient explanation (GRAD), integrated gradients (IG), Smoothgrad (SG), smooth IG (SIG), and XRAI). Ground-truth was defined by the consensus of three fellowship-trained musculoskeletal radiologists who each placed bounding boxes around abnormalities on a holdout saliency test set. Compared to radiologists, all saliency methods showed inferior localization (AUPRCs: 0.438 (SG)-0.590 (XRAI); average radiologist AUPRC: 0.816), repeatability (IoUs: 0.427 (SG)-0.551 (IG); average radiologist IOU: 0.613), and reproducibility (IoUs: 0.250 (SG)-0.502 (XRAI); average radiologist IOU: 0.613) on abnormalities such as fractures, orthopedic hardware insertions, and arthritis. Five methods (GCAM, GRAD, IG, SG, XRAI) passed the sensitivity test. Ultimately, no saliency method met all four trustworthiness criteria; therefore, we recommend caution and rigorous evaluation of saliency maps prior to their clinical use.

Bi-level Graph Learning Unveils Prognosis-Relevant Tumor Microenvironment Patterns from Breast Multiplexed Digital Pathology.

The tumor microenvironment is widely recognized for its central role in driving cancer progression and influencing prognostic outcomes. Despite extensive research efforts dedicated to characterizing this complex and heterogeneous environment, considerable challenges persist. In this study, we introduce a data-driven approach for identifying patterns of cell organizations in the tumor microenvironment that are associated with patient prognoses. Our methodology relies on the construction of a bi-level graph model: (i) a cellular graph, which models the intricate tumor microenvironment, and (ii) a population graph that captures inter-patient similarities, given their respective cellular graphs, by means of a soft Weisfeiler-Lehman subtree kernel. This systematic integration of information across different scales enables us to identify patient subgroups exhibiting unique prognoses while unveiling tumor microenvironment patterns that characterize them. We demonstrate our approach in a cohort of breast cancer patients, where the identified tumor microenvironment patterns result in a risk stratification system that provides complementary, new information with respect to alternative standards. Our results, which are validated in a completely independent cohort, allow for new insights into the prognostic implications of the breast tumor microenvironment, and this methodology could be applied to other cancer types more generally.

Simultaneous multi-transient linear-combination modeling of MRS data improves uncertainty estimation.

PURPOSE: The interest in applying and modeling dynamic MRS has recently grown. Two-dimensional modeling yields advantages for the precision of metabolite estimation in interrelated MRS data. However, it is unknown whether including all transients simultaneously in a 2D model without averaging (presuming a stable signal) performs similarly to one-dimensional (1D) modeling of the averaged spectrum. Therefore, we systematically investigated the accuracy, precision, and uncertainty estimation of both described model approaches. METHODS: Monte Carlo simulations of synthetic MRS data were used to compare the accuracy and uncertainty estimation of simultaneous 2D multitransient linear-combination modeling (LCM) with 1D-LCM of the average. A total of 2,500 data sets per condition with different noise representations of a 64-transient MRS experiment at six signal-to-noise levels for two separate spin systems (scyllo-inositol and gamma-aminobutyric acid) were analyzed. Additional data sets with different levels of noise correlation were also analyzed. Modeling accuracy was assessed by determining the relative bias of the estimated amplitudes against the ground truth, and modeling precision was determined by SDs and Cramér-Rao lower bounds (CRLBs). RESULTS: Amplitude estimates for 1D- and 2D-LCM agreed well and showed a similar level of bias compared with the ground truth. Estimated CRLBs agreed well between both models and with ground-truth CRLBs. For correlated noise, the estimated CRLBs increased with the correlation strength for the 1D-LCM but remained stable for the 2D-LCM. CONCLUSION: Our results indicate that the model performance of 2D multitransient LCM is similar to averaged 1D-LCM. This validation on a simplified scenario serves as a necessary basis for further applications of 2D modeling.

Simultaneous multi-transient linear-combination modeling of MRS data improves uncertainty estimation.

Purpose: The interest in applying and modeling dynamic MRS has recently grown. 2D modeling yields advantages for the precision of metabolite estimation in interrelated MRS data. However, it is unknown whether including all transients simultaneously in a 2D model without averaging (presuming a stable signal) performs similarly to 1D modeling of the averaged spectrum. Therefore, we systematically investigated the accuracy, precision, and uncertainty estimation of both described model approaches. Methods: Monte Carlo simulations of synthetic MRS data were used to compare the accuracy and uncertainty estimation of simultaneous 2D multi-transient LCM with 1D-LCM of the average. 2,500 datasets per condition with different noise representations of a 64-transient MRS experiment at 6 signal-to-noise levels for two separate spin systems (scyllo-inositol and GABA) were analyzed. Additional datasets with different levels of noise correlation were also analyzed. Modeling accuracy was assessed by determining the relative bias of the estimated amplitudes against the ground truth, and modeling precision was determined by standard deviations and Cramér-Rao Lower Bounds (CRLB). Results: Amplitude estimates for 1D- and 2D-LCM agreed well and showed similar level of bias compared to the ground truth. Estimated CRLBs agreed well between both models and with ground truth CRLBs. For correlated noise the estimated CRLBs increased with the correlation strength for the 1D-LCM but remained stable for the 2D-LCM. Conclusion: Our results indicate that the model performance of 2D multi-transient LCM is similar to averaged 1D-LCM. This validation on a simplified scenario serves as necessary basis for further applications of 2D modeling.

Examination-Level Supervision for Deep Learning-based Intracranial Hemorrhage Detection on Head CT Scans.

Purpose To compare the effectiveness of weak supervision (ie, with examination-level labels only) and strong supervision (ie, with image-level labels) in training deep learning models for detection of intracranial hemorrhage (ICH) on head CT scans. Materials and Methods In this retrospective study, an attention-based convolutional neural network was trained with either local (ie, image level) or global (ie, examination level) binary labels on the Radiological Society of North America (RSNA) 2019 Brain CT Hemorrhage Challenge dataset of 21 736 examinations (8876 [40.8%] ICH) and 752 422 images (107 784 [14.3%] ICH). The CQ500 (436 examinations; 212 [48.6%] ICH) and CT-ICH (75 examinations; 36 [48.0%] ICH) datasets were employed for external testing. Performance in detecting ICH was compared between weak (examination-level labels) and strong (image-level labels) learners as a function of the number of labels available during training. Results On examination-level binary classification, strong and weak learners did not have different area under the receiver operating characteristic curve values on the internal validation split (0.96 vs 0.96; P = .64) and the CQ500 dataset (0.90 vs 0.92; P = .15). Weak learners outperformed strong ones on the CT-ICH dataset (0.95 vs 0.92; P = .03). Weak learners had better section-level ICH detection performance when more than 10 000 labels were available for training (average f1 = 0.73 vs 0.65; P < .001). Weakly supervised models trained on the entire RSNA dataset required 35 times fewer labels than equivalent strong learners. Conclusion Strongly supervised models did not achieve better performance than weakly supervised ones, which could reduce radiologist labor requirements for prospective dataset curation. Keywords: CT, Head/Neck, Brain/Brain Stem, Hemorrhage Supplemental material is available for this article. © RSNA, 2023 See also commentary by Wahid and Fuentes in this issue.

The use of weakly supervised machine learning for necrosis assessment in patients with osteosarcoma: A pilot study.

Percent necrosis (PN) following chemotherapy is a prognostic factor for survival in osteosarcoma. Pathologists estimate PN by calculating tumor viability over an average of whole-slide images (WSIs). This non-standardized, labor-intensive process requires specialized training and has high interobserver variability. Therefore, we aimed to develop a machine-learning model capable of calculating PN in osteosarcoma with similar accuracy to that of a musculoskeletal pathologist. In this proof-of-concept study, we retrospectively obtained six WSIs from two patients with conventional osteosarcomas. A weakly supervised learning model was trained by using coarse and incomplete annotations of viable tumor, necrotic tumor, and nontumor tissue in WSIs. Weakly supervised learning refers to processes capable of creating predictive models on the basis of partially and imprecisely annotated data. Once "trained," the model segmented areas of tissue and determined PN of the same six WSIs. To assess model fidelity, the pathologist also estimated PN of each WSI, and we compared the estimates using Pearson's correlation and mean absolute error (MAE). MAE was 15% over the six samples, and 6.4% when an outlier was removed, for which the model inaccurately labeled cartilaginous tissue. The model and pathologist estimates were strongly, positively correlated (r = 0.85). Thus, we created and trained a weakly supervised machine learning model to segment viable tumor, necrotic tumor, and nontumor and to calculate PN with accuracy similar to that of a musculoskeletal pathologist. We expect improvement can be achieved by annotating cartilaginous and other mesenchymal tissue for better representation of the histological heterogeneity in osteosarcoma.

Coarse Race and Ethnicity Labels Mask Granular Underdiagnosis Disparities in Deep Learning Models for Chest Radiograph Diagnosis.

See also the editorial by Nikolic in this issue.

Notice of Removal: Fourier Diffusion Models: A Method to Control MTF and NPS in Score-Based Stochastic Image Generation.

Removed.

Accurate Prediction of Antimicrobial Susceptibility for Point-of-Care Testing of Urine in Less than 90 Minutes via iPRISM Cassettes.

The extensive and improper use of antibiotics has led to a dramatic increase in the frequency of antibiotic resistance among human pathogens, complicating infectious disease treatments. In this work, a method for rapid antimicrobial susceptibility testing (AST) is presented using microstructured silicon diffraction gratings integrated into prototype devices, which enhance bacteria-surface interactions and promote bacterial colonization. The silicon microstructures act also as optical sensors for monitoring bacterial growth upon exposure to antibiotics in a real-time and label-free manner via intensity-based phase-shift reflectometric interference spectroscopic measurements (iPRISM). Rapid AST using clinical isolates of Escherichia coli (E. coli) from urine is established and the assay is applied directly on unprocessed urine samples from urinary tract infection patients. When coupled with a machine learning algorithm trained on clinical samples, the iPRISM AST is able to predict the resistance or susceptibility of a new clinical sample with an Area Under the Receiver Operating Characteristic curve (AUC) of ∼ 0.85 in 1 h, and AUC > 0.9 in 90 min, when compared to state-of-the-art automated AST methods used in the clinic while being an order of magnitude faster.

Tunable neural networks for CT image formation.

Optimization of CT image quality typically involves balancing variance and bias. In traditional filtered back-projection, this trade-off is controlled by the filter cutoff frequency. In model-based iterative reconstruction, the regularization strength parameter often serves the same function. Deep neural networks (DNNs) typically do not provide this tunable control over output image properties. Models are often trained to minimize the expected mean squared error, which penalizes both variance and bias in image outputs but does not offer any control over the trade-off between the two. We propose a method for controlling the output image properties of neural networks with a new loss function called weighted covariance and bias (WCB). Our proposed method uses multiple noise realizations of the input images during training to allow for separate weighting matrices for the variance and bias penalty terms. Moreover, we show that tuning these weights enables targeted penalization of specific image features with spatial frequency domain penalties. To evaluate our method, we present a simulation study using digital anthropomorphic phantoms, physical simulation of CT measurements, and image formation with various algorithms. We show that the WCB loss function offers a greater degree of control over trade-offs between variance and bias, whereas mean-squared error provides only one specific image quality configuration. We also show that WCB can be used to control specific image properties including variance, bias, spatial resolution, and the noise correlation of neural network outputs. Finally, we present a method to optimize the proposed weights for a spiculated lung nodule shape discrimination task. Our results demonstrate this new image quality can control the image properties of DNN outputs and optimize image quality for task-specific performance.

Cross-modality supervised image restoration enables nanoscale tracking of synaptic plasticity in living mice.

Learning is thought to involve changes in glutamate receptors at synapses, submicron structures that mediate communication between neurons in the central nervous system. Due to their small size and high density, synapses are difficult to resolve in vivo, limiting our ability to directly relate receptor dynamics to animal behavior. Here we developed a combination of computational and biological methods to overcome these challenges. First, we trained a deep-learning image-restoration algorithm that combines the advantages of ex vivo super-resolution and in vivo imaging modalities to overcome limitations specific to each optical system. When applied to in vivo images from transgenic mice expressing fluorescently labeled glutamate receptors, this restoration algorithm super-resolved synapses, enabling the tracking of behavior-associated synaptic plasticity with high spatial resolution. This method demonstrates the capabilities of image enhancement to learn from ex vivo data and imaging techniques to improve in vivo imaging resolution.

DeepSTI: Towards tensor reconstruction using fewer orientations in susceptibility tensor imaging.

Susceptibility tensor imaging (STI) is an emerging magnetic resonance imaging technique that characterizes the anisotropic tissue magnetic susceptibility with a second-order tensor model. STI has the potential to provide information for both the reconstruction of white matter fiber pathways and detection of myelin changes in the brain at mm resolution or less, which would be of great value for understanding brain structure and function in healthy and diseased brain. However, the application of STI in vivo has been hindered by its cumbersome and time-consuming acquisition requirement of measuring susceptibility induced MR phase changes at multiple head orientations. Usually, sampling at more than six orientations is required to obtain sufficient information for the ill-posed STI dipole inversion. This complexity is enhanced by the limitation in head rotation angles due to physical constraints of the head coil. As a result, STI has not yet been widely applied in human studies in vivo. In this work, we tackle these issues by proposing an image reconstruction algorithm for STI that leverages data-driven priors. Our method, called DeepSTI, learns the data prior implicitly via a deep neural network that approximates the proximal operator of a regularizer function for STI. The dipole inversion problem is then solved iteratively using the learned proximal network. Experimental results using both simulation and in vivo human data demonstrate great improvement over state-of-the-art algorithms in terms of the reconstructed tensor image, principal eigenvector maps and tractography results, while allowing for tensor reconstruction with MR phase measured at much less than six different orientations. Notably, promising reconstruction results are achieved by our method from only one orientation in human in vivo, and we demonstrate a potential application of this technique for estimating lesion susceptibility anisotropy in patients with multiple sclerosis.

Effect of motion, cortical orientation and spatial resolution on quantitative imaging of cortical R2* and magnetic susceptibility at 0.3 mm in-plane resolution at 7 T.

MR images of the effective relaxation rate R2* and magnetic susceptibility χ derived from multi-echo T2*-weighted (T2*w) MRI can provide insight into iron and myelin distributions in the brain, with the potential of providing biomarkers for neurological disorders. Quantification of R2* and χ at submillimeter resolution in the cortex in vivo has been difficult because of challenges such as head motion, limited signal to noise ratio, long scan time, and motion related magnetic field fluctuations. This work aimed to improve the robustness for quantifying intracortical R2* and χ and analyze the effects from motion, spatial resolution, and cortical orientation. T2*w data was acquired with a spatial resolution of 0.3 × 0.3 × 0.4 mm3 at 7 T and downsampled to various lower resolutions. A combined correction for motion and B0 changes was deployed using volumetric navigators. Such correction improved the T2*w image quality rated by experienced image readers and test-retest reliability of R2* and χ quantification with reduced median inter-scan differences up to 10 s-1 and 5 ppb, respectively. R2* and χ near the line of Gennari, a cortical layer high in iron and myelin, were as much as 10 s-1 and 10 ppb higher than the region at adjacent cortical depth. In addition, a significant effect due to the cortical orientation relative to the static field (B0) was observed in χ with a peak-to-peak amplitude of about 17 ppb. In retrospectively downsampled data, the capability to distinguish different cortical depth regions based on R2* or χ contrast remained up to isotropic 0.5 mm resolution. This study highlights the unique characteristics of R2* and χ along the cortical depth at submillimeter resolution and the need for motion and B0 corrections for their robust quantification in vivo.

Fast Hierarchical Games for Image Explanations.

As modern complex neural networks keep breaking records and solving harder problems, their predictions also become less and less intelligible. The current lack of interpretability often undermines the deployment of accurate machine learning tools in sensitive settings. In this work, we present a model-agnostic explanation method for image classification based on a hierarchical extension of Shapley coefficients-Hierarchical Shap (h-Shap)-that resolves some of the limitations of current approaches. Unlike other Shapley-based explanation methods, h-Shap is scalable and can be computed without the need of approximation. Under certain distributional assumptions, such as those common in multiple instance learning, h-Shap retrieves the exact Shapley coefficients with an exponential improvement in computational complexity. We compare our hierarchical approach with popular Shapley-based and non-Shapley-based methods on a synthetic dataset, a medical imaging scenario, and a general computer vision problem, showing that h-Shap outperforms the state-of-the-art in both accuracy and runtime. Code and experiments are made publicly available.

Estimating and Controlling for Equalized Odds via Sensitive Attribute Predictors.

As the use of machine learning models in real world high-stakes decision settings continues to grow, it is highly important that we are able to audit and control for any potential fairness violations these models may exhibit towards certain groups. To do so, one naturally requires access to sensitive attributes, such as demographics, biological sex, or other potentially sensitive features that determine group membership. Unfortunately, in many settings, this information is often unavailable. In this work we study the well known equalized odds (EOD) definition of fairness. In a setting without sensitive attributes, we first provide tight and computable upper bounds for the EOD violation of a predictor. These bounds precisely reflect the worst possible EOD violation. Second, we demonstrate how one can provably control the worst-case EOD by a new post-processing correction method. Our results characterize when directly controlling for EOD with respect to the predicted sensitive attributes is - and when is not - optimal when it comes to controlling worst-case EOD. Our results hold under assumptions that are milder than previous works, and we illustrate these results with experiments on synthetic and real datasets.

Task-Driven CT Image Quality Optimization for Low-Contrast Lesion Detectability with Tunable Neural Networks.

Low-contrast lesions are difficult to detect in noisy low-dose CT images. Improving CT image quality for this detection task has the potential to improve diagnostic accuracy and patient outcomes. In this work, we use tunable neural networks for CT image restoration with a hyperparameter to control the variance/bias tradeoff. We use clinical images from a super-high-resolution normal-dose CT scan to synthesize low-contrast low-dose CT images for supervised training of deep learning CT reconstruction models. Those models are trained using with multiple noise realizations so that variance and bias can be penalized separately. We use a training loss function with one hyperparameter called the denoising level, which controls the variance/bias tradeoff. Finally, we evaluate the CT image quality to find the optimal denoising level for low-contrast lesion detectability. We evaluate performance using a shallow neural network model classification model to represent a suboptimal image observer. Our results indicate that the optimal networks for low-contrast lesion detectability are those that prioritize bias reduction rather than mean-squared error, which demonstrates the potential clinical benefit of our proposed tunable neural networks.

Visual Transformers and Convolutional Neural Networks for Disease Classification on Radiographs: A Comparison of Performance, Sample Efficiency, and Hidden Stratification.

Purpose: To compare performance, sample efficiency, and hidden stratification of visual transformer (ViT) and convolutional neural network (CNN) architectures for diagnosis of disease on chest radiographs and extremity radiographs using transfer learning. Materials and Methods: In this HIPAA-compliant retrospective study, the authors fine-tuned data-efficient image transformers (DeiT) ViT and CNN classification models pretrained on ImageNet using the National Institutes of Health Chest X-ray 14 dataset (112 120 images) and MURA dataset (14 656 images) for thoracic disease and extremity abnormalities, respectively. Performance was assessed on internal test sets and 75 000 external chest radiographs (three datasets). The primary comparison was DeiT-B ViT vs DenseNet121 CNN; secondary comparisons included DeiT-Ti (Tiny), ResNet152, and EfficientNetB7. Sample efficiency was evaluated by training models on varying dataset sizes. Hidden stratification was evaluated by comparing prevalence of chest tubes in pneumothorax false-positive and false-negative predictions and specific abnormalities for MURA false-negative predictions. Results: DeiT-B weighted area under the receiver operating characteristic curve (wAUC) was slightly lower than that for DenseNet121 on chest radiograph (0.78 vs 0.79; P < .001) and extremity (0.887 vs 0.893; P < .001) internal test sets and chest radiograph external test sets (P < .001 for each). DeiT-B and DeiT-Ti both performed slightly worse than all CNNs for chest radiograph and extremity tasks. DeiT-B and DenseNet121 showed similar sample efficiency. DeiT-B had lower chest tube prevalence in false-positive predictions than DenseNet121 (43.1% [324 of 5088] vs 47.9% [2290 of 4782]). Conclusion: Although DeiT models had lower wAUCs than CNNs for chest radiograph and extremity domains, the differences may be negligible in clinical practice. DeiT-B had sample efficiency similar to that of DenseNet121 and may be less susceptible to certain types of hidden stratification.Keywords: Computer-aided Diagnosis, Informatics, Neural Networks, Thorax, Skeletal-Appendicular, Convolutional Neural Network (CNN), Feature Detection, Supervised Learning, Machine Learning, Deep Learning Supplemental material is available for this article. © RSNA, 2022.

Local Linearity Analysis of Deep Learning CT Denoising Algorithms.

The rapid development of deep-learning methods in medical imaging has called for an analysis method suitable for non-linear and data-dependent algorithms. In this work, we investigate a local linearity analysis where a complex neural network can be represented as piecewise linear systems. We recognize that a large number of neural networks consists of alternating linear layers and rectified linear unit (ReLU) activations, and are therefore strictly piecewise linear. We investigated the extent of these locally linear regions by gradually adding perturbations to an operating point. For this work, we explored perturbations based on image features of interest, including lesion contrast, background, and additive noise. We then developed strategies to extend these strictly locally linear regions to include neighboring linear regions with similar gradients. Using these approximately linear regions, we applied singular value decomposition (SVD) analysis to each local linear system to investigate and explain the overall nonlinear and data-dependent behaviors of neural networks. The analysis was applied to an example CT denoising algorithm trained on thorax CT scans. We observed that the strictly local linear regions are highly sensitive to small signal perturbations. Over a range of lesion contrast from 0.007 to 0.04 mm-1, there is a total of 33992 linear regions. The Jacobians are also shift-variant. However, the Jacobians of neighboring linear regions are very similar. By combining linear regions with similar Jacobians, we narrowed down the number of approximately linear regions to four over lesion contrast from 0.001 to 0.08 mm-1. The SVD analysis to different linear regions revealed denoising behavior that is highly dependent on the background intensity. Analysis further identified greater amount of noise reduction in uniform regions compared to lesion edges. In summary, the local linearity analysis framework we proposed has the potential for us to better characterize and interpret the non-linear and data-dependent behaviors of neural networks.