Cross-modality supervised image restoration enables nanoscale tracking of synaptic plasticity in living mice.

Learning is thought to involve changes in glutamate receptors at synapses, submicron structures that mediate communication between neurons in the central nervous system. Due to their small size and high density, synapses are difficult to resolve in vivo, limiting our ability to directly relate receptor dynamics to animal behavior. Here we developed a combination of computational and biological methods to overcome these challenges. First, we trained a deep-learning image-restoration algorithm that combines the advantages of ex vivo super-resolution and in vivo imaging modalities to overcome limitations specific to each optical system. When applied to in vivo images from transgenic mice expressing fluorescently labeled glutamate receptors, this restoration algorithm super-resolved synapses, enabling the tracking of behavior-associated synaptic plasticity with high spatial resolution. This method demonstrates the capabilities of image enhancement to learn from ex vivo data and imaging techniques to improve in vivo imaging resolution.

Simultaneous multi-transient linear-combination modeling of MRS data improves uncertainty estimation.

PURPOSE: The interest in applying and modeling dynamic MRS has recently grown. Two-dimensional modeling yields advantages for the precision of metabolite estimation in interrelated MRS data. However, it is unknown whether including all transients simultaneously in a 2D model without averaging (presuming a stable signal) performs similarly to one-dimensional (1D) modeling of the averaged spectrum. Therefore, we systematically investigated the accuracy, precision, and uncertainty estimation of both described model approaches. METHODS: Monte Carlo simulations of synthetic MRS data were used to compare the accuracy and uncertainty estimation of simultaneous 2D multitransient linear-combination modeling (LCM) with 1D-LCM of the average. A total of 2,500 data sets per condition with different noise representations of a 64-transient MRS experiment at six signal-to-noise levels for two separate spin systems (scyllo-inositol and gamma-aminobutyric acid) were analyzed. Additional data sets with different levels of noise correlation were also analyzed. Modeling accuracy was assessed by determining the relative bias of the estimated amplitudes against the ground truth, and modeling precision was determined by SDs and Cramér-Rao lower bounds (CRLBs). RESULTS: Amplitude estimates for 1D- and 2D-LCM agreed well and showed a similar level of bias compared with the ground truth. Estimated CRLBs agreed well between both models and with ground-truth CRLBs. For correlated noise, the estimated CRLBs increased with the correlation strength for the 1D-LCM but remained stable for the 2D-LCM. CONCLUSION: Our results indicate that the model performance of 2D multitransient LCM is similar to averaged 1D-LCM. This validation on a simplified scenario serves as a necessary basis for further applications of 2D modeling.

Effect of motion, cortical orientation and spatial resolution on quantitative imaging of cortical R2* and magnetic susceptibility at 0.3 mm in-plane resolution at 7 T.

MR images of the effective relaxation rate R2* and magnetic susceptibility χ derived from multi-echo T2*-weighted (T2*w) MRI can provide insight into iron and myelin distributions in the brain, with the potential of providing biomarkers for neurological disorders. Quantification of R2* and χ at submillimeter resolution in the cortex in vivo has been difficult because of challenges such as head motion, limited signal to noise ratio, long scan time, and motion related magnetic field fluctuations. This work aimed to improve the robustness for quantifying intracortical R2* and χ and analyze the effects from motion, spatial resolution, and cortical orientation. T2*w data was acquired with a spatial resolution of 0.3 × 0.3 × 0.4 mm3 at 7 T and downsampled to various lower resolutions. A combined correction for motion and B0 changes was deployed using volumetric navigators. Such correction improved the T2*w image quality rated by experienced image readers and test-retest reliability of R2* and χ quantification with reduced median inter-scan differences up to 10 s-1 and 5 ppb, respectively. R2* and χ near the line of Gennari, a cortical layer high in iron and myelin, were as much as 10 s-1 and 10 ppb higher than the region at adjacent cortical depth. In addition, a significant effect due to the cortical orientation relative to the static field (B0) was observed in χ with a peak-to-peak amplitude of about 17 ppb. In retrospectively downsampled data, the capability to distinguish different cortical depth regions based on R2* or χ contrast remained up to isotropic 0.5 mm resolution. This study highlights the unique characteristics of R2* and χ along the cortical depth at submillimeter resolution and the need for motion and B0 corrections for their robust quantification in vivo.

Coarse Race and Ethnicity Labels Mask Granular Underdiagnosis Disparities in Deep Learning Models for Chest Radiograph Diagnosis.

See also the editorial by Nikolic in this issue.

Deep learning prediction of sex on chest radiographs: a potential contributor to biased algorithms.

BACKGROUND: Deep convolutional neural networks (DCNNs) for diagnosis of disease on chest radiographs (CXR) have been shown to be biased against males or females if the datasets used to train them have unbalanced sex representation. Prior work has suggested that DCNNs can predict sex on CXR, which could aid forensic evaluations, but also be a source of bias. OBJECTIVE: To (1) evaluate the performance of DCNNs for predicting sex across different datasets and architectures and (2) evaluate visual biomarkers used by DCNNs to predict sex on CXRs. MATERIALS AND METHODS: Chest radiographs were obtained from the Stanford CheXPert and NIH Chest XRay14 datasets which comprised of 224,316 and 112,120 CXRs, respectively. To control for dataset size and class imbalance, random undersampling was used to reduce each dataset to 97,560 images that were balanced for sex. Each dataset was randomly split into training (70%), validation (10%), and test (20%) sets. Four DCNN architectures pre-trained on ImageNet were used for transfer learning. DCNNs were externally validated using a test set from the opposing dataset. Performance was evaluated using area under the receiver operating characteristic curve (AUC). Class activation mapping (CAM) was used to generate heatmaps visualizing the regions contributing to the DCNN's prediction. RESULTS: On the internal test set, DCNNs achieved AUROCs ranging from 0.98 to 0.99. On external validation, the models reached peak cross-dataset performance of 0.94 for the VGG19-Stanford model and 0.95 for the InceptionV3-NIH model. Heatmaps highlighted similar regions of attention between model architectures and datasets, localizing to the mediastinal and upper rib regions, as well as to the lower chest/diaphragmatic regions. CONCLUSION: DCNNs trained on two large CXR datasets accurately predicted sex on internal and external test data with similar heatmap localizations across DCNN architectures and datasets. These findings support the notion that DCNNs can leverage imaging biomarkers to predict sex and potentially confound the accurate prediction of disease on CXRs and contribute to biased models. On the other hand, these DCNNs can be beneficial to emergency radiologists for forensic evaluations and identifying patient sex for patients whose identities are unknown, such as in acute trauma.

Geometric potentials from deep learning improve prediction of CDR H3 loop structures.

MOTIVATION: Antibody structure is largely conserved, except for a complementarity-determining region featuring six variable loops. Five of these loops adopt canonical folds which can typically be predicted with existing methods, while the remaining loop (CDR H3) remains a challenge due to its highly diverse set of observed conformations. In recent years, deep neural networks have proven to be effective at capturing the complex patterns of protein structure. This work proposes DeepH3, a deep residual neural network that learns to predict inter-residue distances and orientations from antibody heavy and light chain sequence. The output of DeepH3 is a set of probability distributions over distances and orientation angles between pairs of residues. These distributions are converted to geometric potentials and used to discriminate between decoy structures produced by RosettaAntibody and predict new CDR H3 loop structures de novo. RESULTS: When evaluated on the Rosetta antibody benchmark dataset of 49 targets, DeepH3-predicted potentials identified better, same and worse structures [measured by root-mean-squared distance (RMSD) from the experimental CDR H3 loop structure] than the standard Rosetta energy function for 33, 6 and 10 targets, respectively, and improved the average RMSD of predictions by 32.1% (1.4 Å). Analysis of individual geometric potentials revealed that inter-residue orientations were more effective than inter-residue distances for discriminating near-native CDR H3 loops. When applied to de novo prediction of CDR H3 loop structures, DeepH3 achieves an average RMSD of 2.2 ± 1.1 Å on the Rosetta antibody benchmark. AVAILABILITY AND IMPLEMENTATION: DeepH3 source code and pre-trained model parameters are freely available at https://github.com/Graylab/deepH3-distances-orientations. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Examination-Level Supervision for Deep Learning-based Intracranial Hemorrhage Detection on Head CT Scans.

Purpose To compare the effectiveness of weak supervision (ie, with examination-level labels only) and strong supervision (ie, with image-level labels) in training deep learning models for detection of intracranial hemorrhage (ICH) on head CT scans. Materials and Methods In this retrospective study, an attention-based convolutional neural network was trained with either local (ie, image level) or global (ie, examination level) binary labels on the Radiological Society of North America (RSNA) 2019 Brain CT Hemorrhage Challenge dataset of 21 736 examinations (8876 [40.8%] ICH) and 752 422 images (107 784 [14.3%] ICH). The CQ500 (436 examinations; 212 [48.6%] ICH) and CT-ICH (75 examinations; 36 [48.0%] ICH) datasets were employed for external testing. Performance in detecting ICH was compared between weak (examination-level labels) and strong (image-level labels) learners as a function of the number of labels available during training. Results On examination-level binary classification, strong and weak learners did not have different area under the receiver operating characteristic curve values on the internal validation split (0.96 vs 0.96; P = .64) and the CQ500 dataset (0.90 vs 0.92; P = .15). Weak learners outperformed strong ones on the CT-ICH dataset (0.95 vs 0.92; P = .03). Weak learners had better section-level ICH detection performance when more than 10 000 labels were available for training (average f1 = 0.73 vs 0.65; P < .001). Weakly supervised models trained on the entire RSNA dataset required 35 times fewer labels than equivalent strong learners. Conclusion Strongly supervised models did not achieve better performance than weakly supervised ones, which could reduce radiologist labor requirements for prospective dataset curation. Keywords: CT, Head/Neck, Brain/Brain Stem, Hemorrhage Supplemental material is available for this article. © RSNA, 2023 See also commentary by Wahid and Fuentes in this issue.

Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses.

Saliency maps are popularly used to "explain" decisions made by modern machine learning models, including deep convolutional neural networks (DCNNs). While the resulting heatmaps purportedly indicate important image features, their "trustworthiness," i.e., utility and robustness, has not been evaluated for musculoskeletal imaging. The purpose of this study was to systematically evaluate the trustworthiness of saliency maps used in disease diagnosis on upper extremity X-ray images. The underlying DCNNs were trained using the Stanford MURA dataset. We studied four trustworthiness criteria-(1) localization accuracy of abnormalities, (2) repeatability, (3) reproducibility, and (4) sensitivity to underlying DCNN weights-across six different gradient-based saliency methods (Grad-CAM (GCAM), gradient explanation (GRAD), integrated gradients (IG), Smoothgrad (SG), smooth IG (SIG), and XRAI). Ground-truth was defined by the consensus of three fellowship-trained musculoskeletal radiologists who each placed bounding boxes around abnormalities on a holdout saliency test set. Compared to radiologists, all saliency methods showed inferior localization (AUPRCs: 0.438 (SG)-0.590 (XRAI); average radiologist AUPRC: 0.816), repeatability (IoUs: 0.427 (SG)-0.551 (IG); average radiologist IOU: 0.613), and reproducibility (IoUs: 0.250 (SG)-0.502 (XRAI); average radiologist IOU: 0.613) on abnormalities such as fractures, orthopedic hardware insertions, and arthritis. Five methods (GCAM, GRAD, IG, SG, XRAI) passed the sensitivity test. Ultimately, no saliency method met all four trustworthiness criteria; therefore, we recommend caution and rigorous evaluation of saliency maps prior to their clinical use.

Bi-level Graph Learning Unveils Prognosis-Relevant Tumor Microenvironment Patterns from Breast Multiplexed Digital Pathology.

The tumor microenvironment is widely recognized for its central role in driving cancer progression and influencing prognostic outcomes. Despite extensive research efforts dedicated to characterizing this complex and heterogeneous environment, considerable challenges persist. In this study, we introduce a data-driven approach for identifying patterns of cell organizations in the tumor microenvironment that are associated with patient prognoses. Our methodology relies on the construction of a bi-level graph model: (i) a cellular graph, which models the intricate tumor microenvironment, and (ii) a population graph that captures inter-patient similarities, given their respective cellular graphs, by means of a soft Weisfeiler-Lehman subtree kernel. This systematic integration of information across different scales enables us to identify patient subgroups exhibiting unique prognoses while unveiling tumor microenvironment patterns that characterize them. We demonstrate our approach in a cohort of breast cancer patients, where the identified tumor microenvironment patterns result in a risk stratification system that provides complementary, new information with respect to alternative standards. Our results, which are validated in a completely independent cohort, allow for new insights into the prognostic implications of the breast tumor microenvironment, and this methodology could be applied to other cancer types more generally.

The use of weakly supervised machine learning for necrosis assessment in patients with osteosarcoma: A pilot study.

Percent necrosis (PN) following chemotherapy is a prognostic factor for survival in osteosarcoma. Pathologists estimate PN by calculating tumor viability over an average of whole-slide images (WSIs). This non-standardized, labor-intensive process requires specialized training and has high interobserver variability. Therefore, we aimed to develop a machine-learning model capable of calculating PN in osteosarcoma with similar accuracy to that of a musculoskeletal pathologist. In this proof-of-concept study, we retrospectively obtained six WSIs from two patients with conventional osteosarcomas. A weakly supervised learning model was trained by using coarse and incomplete annotations of viable tumor, necrotic tumor, and nontumor tissue in WSIs. Weakly supervised learning refers to processes capable of creating predictive models on the basis of partially and imprecisely annotated data. Once "trained," the model segmented areas of tissue and determined PN of the same six WSIs. To assess model fidelity, the pathologist also estimated PN of each WSI, and we compared the estimates using Pearson's correlation and mean absolute error (MAE). MAE was 15% over the six samples, and 6.4% when an outlier was removed, for which the model inaccurately labeled cartilaginous tissue. The model and pathologist estimates were strongly, positively correlated (r = 0.85). Thus, we created and trained a weakly supervised machine learning model to segment viable tumor, necrotic tumor, and nontumor and to calculate PN with accuracy similar to that of a musculoskeletal pathologist. We expect improvement can be achieved by annotating cartilaginous and other mesenchymal tissue for better representation of the histological heterogeneity in osteosarcoma.

Simultaneous multi-transient linear-combination modeling of MRS data improves uncertainty estimation.

Purpose: The interest in applying and modeling dynamic MRS has recently grown. 2D modeling yields advantages for the precision of metabolite estimation in interrelated MRS data. However, it is unknown whether including all transients simultaneously in a 2D model without averaging (presuming a stable signal) performs similarly to 1D modeling of the averaged spectrum. Therefore, we systematically investigated the accuracy, precision, and uncertainty estimation of both described model approaches. Methods: Monte Carlo simulations of synthetic MRS data were used to compare the accuracy and uncertainty estimation of simultaneous 2D multi-transient LCM with 1D-LCM of the average. 2,500 datasets per condition with different noise representations of a 64-transient MRS experiment at 6 signal-to-noise levels for two separate spin systems (scyllo-inositol and GABA) were analyzed. Additional datasets with different levels of noise correlation were also analyzed. Modeling accuracy was assessed by determining the relative bias of the estimated amplitudes against the ground truth, and modeling precision was determined by standard deviations and Cramér-Rao Lower Bounds (CRLB). Results: Amplitude estimates for 1D- and 2D-LCM agreed well and showed similar level of bias compared to the ground truth. Estimated CRLBs agreed well between both models and with ground truth CRLBs. For correlated noise the estimated CRLBs increased with the correlation strength for the 1D-LCM but remained stable for the 2D-LCM. Conclusion: Our results indicate that the model performance of 2D multi-transient LCM is similar to averaged 1D-LCM. This validation on a simplified scenario serves as necessary basis for further applications of 2D modeling.

Fast Hierarchical Games for Image Explanations.

As modern complex neural networks keep breaking records and solving harder problems, their predictions also become less and less intelligible. The current lack of interpretability often undermines the deployment of accurate machine learning tools in sensitive settings. In this work, we present a model-agnostic explanation method for image classification based on a hierarchical extension of Shapley coefficients-Hierarchical Shap (h-Shap)-that resolves some of the limitations of current approaches. Unlike other Shapley-based explanation methods, h-Shap is scalable and can be computed without the need of approximation. Under certain distributional assumptions, such as those common in multiple instance learning, h-Shap retrieves the exact Shapley coefficients with an exponential improvement in computational complexity. We compare our hierarchical approach with popular Shapley-based and non-Shapley-based methods on a synthetic dataset, a medical imaging scenario, and a general computer vision problem, showing that h-Shap outperforms the state-of-the-art in both accuracy and runtime. Code and experiments are made publicly available.

Estimating and Controlling for Equalized Odds via Sensitive Attribute Predictors.

As the use of machine learning models in real world high-stakes decision settings continues to grow, it is highly important that we are able to audit and control for any potential fairness violations these models may exhibit towards certain groups. To do so, one naturally requires access to sensitive attributes, such as demographics, biological sex, or other potentially sensitive features that determine group membership. Unfortunately, in many settings, this information is often unavailable. In this work we study the well known equalized odds (EOD) definition of fairness. In a setting without sensitive attributes, we first provide tight and computable upper bounds for the EOD violation of a predictor. These bounds precisely reflect the worst possible EOD violation. Second, we demonstrate how one can provably control the worst-case EOD by a new post-processing correction method. Our results characterize when directly controlling for EOD with respect to the predicted sensitive attributes is - and when is not - optimal when it comes to controlling worst-case EOD. Our results hold under assumptions that are milder than previous works, and we illustrate these results with experiments on synthetic and real datasets.

DeepSTI: Towards tensor reconstruction using fewer orientations in susceptibility tensor imaging.

Susceptibility tensor imaging (STI) is an emerging magnetic resonance imaging technique that characterizes the anisotropic tissue magnetic susceptibility with a second-order tensor model. STI has the potential to provide information for both the reconstruction of white matter fiber pathways and detection of myelin changes in the brain at mm resolution or less, which would be of great value for understanding brain structure and function in healthy and diseased brain. However, the application of STI in vivo has been hindered by its cumbersome and time-consuming acquisition requirement of measuring susceptibility induced MR phase changes at multiple head orientations. Usually, sampling at more than six orientations is required to obtain sufficient information for the ill-posed STI dipole inversion. This complexity is enhanced by the limitation in head rotation angles due to physical constraints of the head coil. As a result, STI has not yet been widely applied in human studies in vivo. In this work, we tackle these issues by proposing an image reconstruction algorithm for STI that leverages data-driven priors. Our method, called DeepSTI, learns the data prior implicitly via a deep neural network that approximates the proximal operator of a regularizer function for STI. The dipole inversion problem is then solved iteratively using the learned proximal network. Experimental results using both simulation and in vivo human data demonstrate great improvement over state-of-the-art algorithms in terms of the reconstructed tensor image, principal eigenvector maps and tractography results, while allowing for tensor reconstruction with MR phase measured at much less than six different orientations. Notably, promising reconstruction results are achieved by our method from only one orientation in human in vivo, and we demonstrate a potential application of this technique for estimating lesion susceptibility anisotropy in patients with multiple sclerosis.

Tunable neural networks for CT image formation.

Optimization of CT image quality typically involves balancing variance and bias. In traditional filtered back-projection, this trade-off is controlled by the filter cutoff frequency. In model-based iterative reconstruction, the regularization strength parameter often serves the same function. Deep neural networks (DNNs) typically do not provide this tunable control over output image properties. Models are often trained to minimize the expected mean squared error, which penalizes both variance and bias in image outputs but does not offer any control over the trade-off between the two. We propose a method for controlling the output image properties of neural networks with a new loss function called weighted covariance and bias (WCB). Our proposed method uses multiple noise realizations of the input images during training to allow for separate weighting matrices for the variance and bias penalty terms. Moreover, we show that tuning these weights enables targeted penalization of specific image features with spatial frequency domain penalties. To evaluate our method, we present a simulation study using digital anthropomorphic phantoms, physical simulation of CT measurements, and image formation with various algorithms. We show that the WCB loss function offers a greater degree of control over trade-offs between variance and bias, whereas mean-squared error provides only one specific image quality configuration. We also show that WCB can be used to control specific image properties including variance, bias, spatial resolution, and the noise correlation of neural network outputs. Finally, we present a method to optimize the proposed weights for a spiculated lung nodule shape discrimination task. Our results demonstrate this new image quality can control the image properties of DNN outputs and optimize image quality for task-specific performance.

Task-Driven CT Image Quality Optimization for Low-Contrast Lesion Detectability with Tunable Neural Networks.

Low-contrast lesions are difficult to detect in noisy low-dose CT images. Improving CT image quality for this detection task has the potential to improve diagnostic accuracy and patient outcomes. In this work, we use tunable neural networks for CT image restoration with a hyperparameter to control the variance/bias tradeoff. We use clinical images from a super-high-resolution normal-dose CT scan to synthesize low-contrast low-dose CT images for supervised training of deep learning CT reconstruction models. Those models are trained using with multiple noise realizations so that variance and bias can be penalized separately. We use a training loss function with one hyperparameter called the denoising level, which controls the variance/bias tradeoff. Finally, we evaluate the CT image quality to find the optimal denoising level for low-contrast lesion detectability. We evaluate performance using a shallow neural network model classification model to represent a suboptimal image observer. Our results indicate that the optimal networks for low-contrast lesion detectability are those that prioritize bias reduction rather than mean-squared error, which demonstrates the potential clinical benefit of our proposed tunable neural networks.

Notice of Removal: Fourier Diffusion Models: A Method to Control MTF and NPS in Score-Based Stochastic Image Generation.

Removed.

Accurate Prediction of Antimicrobial Susceptibility for Point-of-Care Testing of Urine in Less than 90 Minutes via iPRISM Cassettes.

The extensive and improper use of antibiotics has led to a dramatic increase in the frequency of antibiotic resistance among human pathogens, complicating infectious disease treatments. In this work, a method for rapid antimicrobial susceptibility testing (AST) is presented using microstructured silicon diffraction gratings integrated into prototype devices, which enhance bacteria-surface interactions and promote bacterial colonization. The silicon microstructures act also as optical sensors for monitoring bacterial growth upon exposure to antibiotics in a real-time and label-free manner via intensity-based phase-shift reflectometric interference spectroscopic measurements (iPRISM). Rapid AST using clinical isolates of Escherichia coli (E. coli) from urine is established and the assay is applied directly on unprocessed urine samples from urinary tract infection patients. When coupled with a machine learning algorithm trained on clinical samples, the iPRISM AST is able to predict the resistance or susceptibility of a new clinical sample with an Area Under the Receiver Operating Characteristic curve (AUC) of ∼ 0.85 in 1 h, and AUC > 0.9 in 90 min, when compared to state-of-the-art automated AST methods used in the clinic while being an order of magnitude faster.

Antibody structure prediction using interpretable deep learning.

Therapeutic antibodies make up a rapidly growing segment of the biologics market. However, rational design of antibodies is hindered by reliance on experimental methods for determining antibody structures. Here, we present DeepAb, a deep learning method for predicting accurate antibody FV structures from sequence. We evaluate DeepAb on a set of structurally diverse, therapeutically relevant antibodies and find that our method consistently outperforms the leading alternatives. Previous deep learning methods have operated as "black boxes" and offered few insights into their predictions. By introducing a directly interpretable attention mechanism, we show our network attends to physically important residue pairs (e.g., proximal aromatics and key hydrogen bonding interactions). Finally, we present a novel mutant scoring metric derived from network confidence and show that for a particular antibody, all eight of the top-ranked mutations improve binding affinity. This model will be useful for a broad range of antibody prediction and design tasks.