Deceptive learning in histopathology.

AIMS: Deep learning holds immense potential for histopathology, automating tasks that are simple for expert pathologists and revealing novel biology for tasks that were previously considered difficult or impossible to solve by eye alone. However, the extent to which the visual strategies learned by deep learning models in histopathological analysis are trustworthy or not has yet to be systematically analysed. Here, we systematically evaluate deep neural networks (DNNs) trained for histopathological analysis in order to understand if their learned strategies are trustworthy or deceptive. METHODS AND RESULTS: We trained a variety of DNNs on a novel data set of 221 whole-slide images (WSIs) from lung adenocarcinoma patients, and evaluated their effectiveness at (1) molecular profiling of KRAS versus EGFR mutations, (2) determining the primary tissue of a tumour and (3) tumour detection. While DNNs achieved above-chance performance on molecular profiling, they did so by exploiting correlations between histological subtypes and mutations, and failed to generalise to a challenging test set obtained through laser capture microdissection (LCM). In contrast, DNNs learned robust and trustworthy strategies for determining the primary tissue of a tumour as well as detecting and localising tumours in tissue. CONCLUSIONS: Our work demonstrates that DNNs hold immense promise for aiding pathologists in analysing tissue. However, they are also capable of achieving seemingly strong performance by learning deceptive strategies that leverage spurious correlations, and are ultimately unsuitable for research or clinical work. The framework we propose for model evaluation and interpretation is an important step towards developing reliable automated systems for histopathological analysis.

Fixing the problems of deep neural networks will require better training data and learning algorithms.

Bowers et al. argue that deep neural networks (DNNs) are poor models of biological vision because they often learn to rival human accuracy by relying on strategies that differ markedly from those of humans. We show that this problem is worsening as DNNs are becoming larger-scale and increasingly more accurate, and prescribe methods for building DNNs that can reliably model biological vision.

Understanding the Computational Demands Underlying Visual Reasoning.

Visual understanding requires comprehending complex visual relations between objects within a scene. Here, we seek to characterize the computational demands for abstract visual reasoning. We do this by systematically assessing the ability of modern deep convolutional neural networks (CNNs) to learn to solve the synthetic visual reasoning test (SVRT) challenge, a collection of 23 visual reasoning problems. Our analysis reveals a novel taxonomy of visual reasoning tasks, which can be primarily explained by both the type of relations (same-different versus spatial-relation judgments) and the number of relations used to compose the underlying rules. Prior cognitive neuroscience work suggests that attention plays a key role in humans' visual reasoning ability. To test this hypothesis, we extended the CNNs with spatial and feature-based attention mechanisms. In a second series of experiments, we evaluated the ability of these attention networks to learn to solve the SVRT challenge and found the resulting architectures to be much more efficient at solving the hardest of these visual reasoning tasks. Most important, the corresponding improvements on individual tasks partially explained our novel taxonomy. Overall, this work provides a granular computational account of visual reasoning and yields testable neuroscience predictions regarding the differential need for feature-based versus spatial attention depending on the type of visual reasoning problem.

Congenital myopathy results from misregulation of a muscle Ca2+ channel by mutant Stac3.

Skeletal muscle contractions are initiated by an increase in Ca2+ released during excitation-contraction (EC) coupling, and defects in EC coupling are associated with human myopathies. EC coupling requires communication between voltage-sensing dihydropyridine receptors (DHPRs) in transverse tubule membrane and Ca2+ release channel ryanodine receptor 1 (RyR1) in the sarcoplasmic reticulum (SR). Stac3 protein (SH3 and cysteine-rich domain 3) is an essential component of the EC coupling apparatus and a mutation in human STAC3 causes the debilitating Native American myopathy (NAM), but the nature of how Stac3 acts on the DHPR and/or RyR1 is unknown. Using electron microscopy, electrophysiology, and dynamic imaging of zebrafish muscle fibers, we find significantly reduced DHPR levels, functionality, and stability in stac3 mutants. Furthermore, stac3NAM myofibers exhibited increased caffeine-induced Ca2+ release across a wide range of concentrations in the absence of altered caffeine sensitivity as well as increased Ca2+ in internal stores, which is consistent with increased SR luminal Ca2+ These findings define critical roles for Stac3 in EC coupling and human disease.

Encoding-Stage Crosstalk Between Object- and Spatial Property-Based Scene Processing Pathways.

Scene categorization draws on 2 information sources: The identities of objects scenes contain and scenes' intrinsic spatial properties. Because these resources are formally independent, it is possible for them to leads to conflicting judgments of scene category. We tested the hypothesis that the potential for such conflicts is mitigated by a system of "crosstalk" between object- and spatial layout-processing pathways, under which the encoded spatial properties of scenes are biased by scenes' object contents. Specifically, we show that the presence of objects strongly associated with a given scene category can bias the encoded spatial properties of scenes containing them toward the average of that category, an effect which is evident both in behavioral measures of scenes' perceived spatial properties and in scene-evoked multivoxel patterns recorded with functional magnetic resonance imaging from the parahippocampal place area (PPA), a region associated with the processing of scenes' spatial properties. These results indicate that harmonization of object- and spatial property-based estimates of scene identity begins when spatial properties are encoded, and that the PPA plays a central role in this process.

Evidence for participation by object-selective visual cortex in scene category judgments.

Scene recognition is a core function of the visual system, drawing both on scenes' intrinsic global features, prominently their spatial properties, and on the identities of the objects scenes contain. Neuroimaging and neuropsychological studies have associated spatial property-based scene categorization with parahippocampal cortex, while processing of scene-relevant object information is associated with the lateral occipital complex (LOC), wherein activity patterns distinguish between categories of standalone objects and those embedded in scenes. However, despite the importance of objects to scene categorization and the role of LOC in processing them, damage or disruption to LOC that hampers object recognition has been shown to improve scene categorization. To address this paradox, we used functional magnetic resonance imaging (fMRI) to directly assess the contributions of LOC and the parahippocampal place area (PPA) to category judgments of indoor scenes that were devoid of objective identity signals. Observers were alternately cued to base judgments on scenes' objects or spatial properties. In both LOC and PPA, multivoxel activity patterns better decoded judgments based on their typically associated features: LOC more accurately decoded object-based judgments, while PPA more accurately decoded spatial property-based judgments. The cue contingency of LOC decoding accuracy indicates that it was not an outcome of feedback from judgments and is instead consistent with dependency of judgments on the output of object processing pathways in which LOC participates.

Harmonizing the object recognition strategies of deep neural networks with humans.

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition. State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves. We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns DNN and human visual strategies and improves categorization accuracy. Our work represents the first demonstration that the scaling laws [1-3] that are guiding the design of DNNs today have also produced worse models of human vision. We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

Fluorescently labeled nuclear morphology is highly informative of neurotoxicity.

Neurotoxicity can be detected in live microscopy by morphological changes such as retraction of neurites, fragmentation, blebbing of the neuronal soma and ultimately the disappearance of fluorescently labeled neurons. However, quantification of these features is often difficult, low-throughput, and imprecise due to the overreliance on human curation. Recently, we showed that convolutional neural network (CNN) models can outperform human curators in the assessment of neuronal death from images of fluorescently labeled neurons, suggesting that there is information within the images that indicates toxicity but that is not apparent to the human eye. In particular, the CNN's decision strategy indicated that information within the nuclear region was essential for its superhuman performance. Here, we systematically tested this prediction by comparing images of fluorescent neuronal morphology from nuclear-localized fluorescent protein to those from freely diffused fluorescent protein for classifying neuronal death. We found that biomarker-optimized (BO-) CNNs could learn to classify neuronal death from fluorescent protein-localized nuclear morphology (mApple-NLS-CNN) alone, with super-human accuracy. Furthermore, leveraging methods from explainable artificial intelligence, we identified novel features within the nuclear-localized fluorescent protein signal that were indicative of neuronal death. Our findings suggest that the use of a nuclear morphology marker in live imaging combined with computational models such mApple-NLS-CNN can provide an optimal readout of neuronal death, a common result of neurotoxicity.

Development of a Deep Learning Algorithm for the Histopathologic Diagnosis and Gleason Grading of Prostate Cancer Biopsies: A Pilot Study.

BACKGROUND: The pathologic diagnosis and Gleason grading of prostate cancer are time-consuming, error-prone, and subject to interobserver variability. Machine learning offers opportunities to improve the diagnosis, risk stratification, and prognostication of prostate cancer. OBJECTIVE: To develop a state-of-the-art deep learning algorithm for the histopathologic diagnosis and Gleason grading of prostate biopsy specimens. DESIGN, SETTING, AND PARTICIPANTS: A total of 85 prostate core biopsy specimens from 25 patients were digitized at 20× magnification and annotated for Gleason 3, 4, and 5 prostate adenocarcinoma by a urologic pathologist. From these virtual slides, we sampled 14803 image patches of 256×256 pixels, approximately balanced for malignancy. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: We trained and tested a deep residual convolutional neural network to classify each patch at two levels: (1) coarse (benign vs malignant) and (2) fine (benign vs Gleason 3 vs 4 vs 5). Model performance was evaluated using fivefold cross-validation. Randomization tests were used for hypothesis testing of model performance versus chance. RESULTS AND LIMITATIONS: The model demonstrated 91.5% accuracy (p<0.001) at coarse-level classification of image patches as benign versus malignant (0.93 sensitivity, 0.90 specificity, and 0.95 average precision). The model demonstrated 85.4% accuracy (p<0.001) at fine-level classification of image patches as benign versus Gleason 3 versus Gleason 4 versus Gleason 5 (0.83 sensitivity, 0.94 specificity, and 0.83 average precision), with the greatest number of confusions in distinguishing between Gleason 3 and 4, and between Gleason 4 and 5. Limitations include the small sample size and the need for external validation. CONCLUSIONS: In this study, a deep learning-based computer vision algorithm demonstrated excellent performance for the histopathologic diagnosis and Gleason grading of prostate cancer. PATIENT SUMMARY: We developed a deep learning algorithm that demonstrated excellent performance for the diagnosis and grading of prostate cancer.

Superhuman cell death detection with biomarker-optimized neural networks.

Cellular events underlying neurodegenerative disease may be captured by longitudinal live microscopy of neurons. While the advent of robot-assisted microscopy has helped scale such efforts to high-throughput regimes with the statistical power to detect transient events, time-intensive human annotation is required. We addressed this fundamental limitation with biomarker-optimized convolutional neural networks (BO-CNNs): interpretable computer vision models trained directly on biosensor activity. We demonstrate the ability of BO-CNNs to detect cell death, which is typically measured by trained annotators. BO-CNNs detected cell death with superhuman accuracy and speed by learning to identify subcellular morphology associated with cell vitality, despite receiving no explicit supervision to rely on these features. These models also revealed an intranuclear morphology signal that is difficult to spot by eye and had not previously been linked to cell death, but that reliably indicates death. BO-CNNs are broadly useful for analyzing live microscopy and essential for interpreting high-throughput experiments.

Small-Molecule Modulation of TDP-43 Recruitment to Stress Granules Prevents Persistent TDP-43 Accumulation in ALS/FTD.

Stress granules (SGs) form during cellular stress and are implicated in neurodegenerative diseases such as amyotrophic lateral sclerosis and frontotemporal dementia (ALS/FTD). To yield insights into the role of SGs in pathophysiology, we performed a high-content screen to identify small molecules that alter SG properties in proliferative cells and human iPSC-derived motor neurons (iPS-MNs). One major class of active molecules contained extended planar aromatic moieties, suggesting a potential to intercalate in nucleic acids. Accordingly, we show that several hit compounds can prevent the RNA-dependent recruitment of the ALS-associated RNA-binding proteins (RBPs) TDP-43, FUS, and HNRNPA2B1 into SGs. We further demonstrate that transient SG formation contributes to persistent accumulation of TDP-43 into cytoplasmic puncta and that our hit compounds can reduce this accumulation in iPS-MNs from ALS patients. We propose that compounds with planar moieties represent a promising starting point to develop small-molecule therapeutics for treating ALS/FTD.

Complementary surrounds explain diverse contextual phenomena across visual modalities.

Context is known to affect how a stimulus is perceived. A variety of illusions have been attributed to contextual processing-from orientation tilt effects to chromatic induction phenomena, but their neural underpinnings remain poorly understood. Here, we present a recurrent network model of classical and extraclassical receptive fields that is constrained by the anatomy and physiology of the visual cortex. A key feature of the model is the postulated existence of near- versus far- extraclassical regions with complementary facilitatory and suppressive contributions to the classical receptive field. The model accounts for a variety of contextual illusions, reveals commonalities between seemingly disparate phenomena, and helps organize them into a novel taxonomy. It explains how center-surround interactions may shift from attraction to repulsion in tilt effects, and from contrast to assimilation in induction phenomena. The model further explains enhanced perceptual shifts generated by a class of patterned background stimuli that activate the two opponent extraclassical regions cooperatively. Overall, the ability of the model to account for the variety and complexity of contextual illusions provides computational evidence for a novel canonical circuit that is shared across visual modalities. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Reliability and validity of the Virtual Reality Lateralized Attention Test in assessing hemispatial neglect in right-hemisphere stroke.

OBJECTIVE: Many tests of hemispatial neglect are insensitive to subtle (but clinically relevant) forms of the disorder. This study provides additional reliability and validity data on the Virtual Reality Lateralized Attention Test (VRLAT), an easy-to-administer computerized measure of hemispatial neglect that mimics the attentional demands of real-world tasks, and has previously shown strong validity and sensitivity (Dawson, Buxbaum, & Rizzo, 2008). The present study assessed a large sample of patients with the aim of developing a clinically useful version of the test, and established the concurrent criterion validity of the VRLAT as measured by its association with performance in a real-world task. METHOD: Seventy consecutively recruited right-hemisphere stroke patients were assessed with the VRLAT, which requires participants to name objects as they navigate (or are navigated) along a winding virtual path. They also performed a real-world navigation task, tests of sensory and motor function, and paper-and-pencil neglect tests. RESULTS: The VRLAT demonstrated strong sensitivity and specificity, minimal practice effects, and strong validity, and outperformed traditional paper-and-pencil tests in the prediction of real-world collisions. CONCLUSIONS: The VRLAT is a sensitive, valid, and reliable measure of hemispatial neglect that requires no specialized equipment, is easy to administer, and is useful for both clinical and research purposes. Moreover, a shortened version with a 5-min administration time has many of the desirable psychometric properties of the original full-length task.