Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians.

Predictive artificial intelligence (AI) systems based on deep learning have been shown to achieve expert-level identification of diseases in multiple medical imaging settings, but can make errors in cases accurately diagnosed by clinicians and vice versa. We developed Complementarity-Driven Deferral to Clinical Workflow (CoDoC), a system that can learn to decide between the opinion of a predictive AI model and a clinical workflow. CoDoC enhances accuracy relative to clinician-only or AI-only baselines in clinical workflows that screen for breast cancer or tuberculosis (TB). For breast cancer screening, compared to double reading with arbitration in a screening program in the UK, CoDoC reduced false positives by 25% at the same false-negative rate, while achieving a 66% reduction in clinician workload. For TB triaging, compared to standalone AI and clinical workflows, CoDoC achieved a 5-15% reduction in false positives at the same false-negative rate for three of five commercially available predictive AI systems. To facilitate the deployment of CoDoC in novel futuristic clinical settings, we present results showing that CoDoC's performance gains are sustained across several axes of variation (imaging modality, clinical setting and predictive AI system) and discuss the limitations of our evaluation and where further validation would be needed. We provide an open-source implementation to encourage further research and application.

Detecting shortcut learning for fair medical AI using shortcut testing.

Machine learning (ML) holds great promise for improving healthcare, but it is critical to ensure that its use will not propagate or amplify health disparities. An important step is to characterize the (un)fairness of ML models-their tendency to perform differently across subgroups of the population-and to understand its underlying mechanisms. One potential driver of algorithmic unfairness, shortcut learning, arises when ML models base predictions on improper correlations in the training data. Diagnosing this phenomenon is difficult as sensitive attributes may be causally linked with disease. Using multitask learning, we propose a method to directly test for the presence of shortcut learning in clinical ML systems and demonstrate its application to clinical tasks in radiology and dermatology. Finally, our approach reveals instances when shortcutting is not responsible for unfairness, highlighting the need for a holistic approach to fairness mitigation in medical AI.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Does your dermatology classifier know what it doesn't know? Detecting the long-tail of unseen conditions.

Supervised deep learning models have proven to be highly effective in classification of dermatological conditions. These models rely on the availability of abundant labeled training examples. However, in the real-world, many dermatological conditions are individually too infrequent for per-condition classification with supervised learning. Although individually infrequent, these conditions may collectively be common and therefore are clinically significant in aggregate. To prevent models from generating erroneous outputs on such examples, there remains a considerable unmet need for deep learning systems that can better detect such infrequent conditions. These infrequent 'outlier' conditions are seen very rarely (or not at all) during training. In this paper, we frame this task as an out-of-distribution (OOD) detection problem. We set up a benchmark ensuring that outlier conditions are disjoint between the model training, validation, and test sets. Unlike traditional OOD detection benchmarks where the task is to detect dataset distribution shift, we aim at the more challenging task of detecting subtle differences resulting from a different pathology or condition. We propose a novel hierarchical outlier detection (HOD) loss, which assigns multiple abstention classes corresponding to each training outlier class and jointly performs a coarse classification of inliers vs. outliers, along with fine-grained classification of the individual classes. We demonstrate that the proposed HOD loss based approach outperforms leading methods that leverage outlier data during training. Further, performance is significantly boosted by using recent representation learning methods (BiT, SimCLR, MICLe). Further, we explore ensembling strategies for OOD detection and propose a diverse ensemble selection process for the best result. We also perform a subgroup analysis over conditions of varying risk levels and different skin types to investigate how OOD performance changes over each subgroup and demonstrate the gains of our framework in comparison to baseline. Furthermore, we go beyond traditional performance metrics and introduce a cost matrix for model trust analysis to approximate downstream clinical impact. We use this cost matrix to compare the proposed method against the baseline, thereby making a stronger case for its effectiveness in real-world scenarios.

Effects of cannabidiol on brain excitation and inhibition systems; a randomised placebo-controlled single dose trial during magnetic resonance spectroscopy in adults with and without autism spectrum disorder.

There is increasing interest in the use of cannabis and its major non-intoxicating component cannabidiol (CBD) as a treatment for mental health and neurodevelopmental disorders, such as autism spectrum disorder (ASD). However, before launching large-scale clinical trials, a better understanding of the effects of CBD on brain would be desirable. Preclinical evidence suggests that one aspect of the polypharmacy of CBD is that it modulates brain excitatory glutamate and inhibitory γ-aminobutyric acid (GABA) levels, including in brain regions linked to ASD, such as the basal ganglia (BG) and the dorsomedial prefrontal cortex (DMPFC). However, differences in glutamate and GABA pathways in ASD mean that the response to CBD in people with and without ASD may be not be the same. To test whether CBD 'shifts' glutamate and GABA levels; and to examine potential differences in this response in ASD, we used magnetic resonance spectroscopy (MRS) to measure glutamate (Glx = glutamate + glutamine) and GABA+ (GABA + macromolecules) levels in 34 healthy men (17 neurotypicals, 17 ASD). Data acquisition commenced 2 h (peak plasma levels) after a single oral dose of 600 mg CBD or placebo. Test sessions were at least 13 days apart. Across groups, CBD increased subcortical, but decreased cortical, Glx. Across regions, CBD increased GABA+ in controls, but decreased GABA+ in ASD; the group difference in change in GABA + in the DMPFC was significant. Thus, CBD modulates glutamate-GABA systems, but prefrontal-GABA systems respond differently in ASD. Our results do not speak to the efficacy of CBD. Future studies should examine the effects of chronic administration on brain and behaviour, and whether acute brain changes predict longer-term response.

Typical magnitude and spatial extent of crowding in autism.

Enhanced spatial processing of local visual details has been reported in individuals with autism spectrum conditions (ASC), and crowding is postulated to be a mechanism that may produce this ability. However, evidence for atypical crowding in ASC is mixed, with some studies reporting a complete lack of crowding in autism and others reporting a typical magnitude of crowding between individuals with and without ASC. Here, we aim to disambiguate these conflicting results by testing both the magnitude and the spatial extent of crowding in individuals with ASC (N = 25) and age- and IQ-matched controls (N = 23) during an orientation discrimination task. We find a strong crowding effect in individuals with and without ASC, which falls off as the distance between target and flanker is increased. Both the magnitude and the spatial range of this effect were comparable between individuals with and without ASC. We also find typical (uncrowded) orientation discrimination thresholds in individuals with ASC. These findings suggest that the spatial extent of crowding is unremarkable in ASC, and is therefore unlikely to account for the visual symptoms reported in individuals with the diagnosis.

Reduced perceptual exclusivity during object and grating rivalry in autism.

The dynamics of binocular rivalry may be a behavioral footprint of excitatory and inhibitory neural transmission in visual cortex. Given the presence of atypical visual features in Autism Spectrum Conditions (ASC), and the growing evidence in support of the idea of an imbalance in excitatory/inhibitory neural transmission in animal and genetic models of ASC, we hypothesized that binocular rivalry might prove a simple behavioral marker of such a transmission imbalance in the autistic brain. In support of this hypothesis, we previously reported a slower rate of rivalry in ASC, driven by longer transitional states between dominant percepts. We tested whether atypical dynamics of binocular rivalry in ASC are specific to certain stimulus features. 53 participants (26 with ASC, matched for age, sex, and IQ) participated in a binocular rivalry experiment in which the dynamics of rivalry were measured at two levels of stimulus complexity, low (grayscale gratings) and high (colored objects). Individuals with ASC experienced a slower rate of binocular rivalry, driven by longer transitional states between dominant percepts. These exaggerated transitional states were present at both low and high levels of stimulus complexity (gratings and objects), suggesting that atypical binocular dynamics in autism are robust with respect to stimulus choice. Interactions between stimulus properties and rivalry dynamics in autism indicate that achromatic grating stimuli produce stronger group differences. These results confirm the finding of atypical dynamics of binocular rivalry in ASC. These dynamics were present for stimuli of both low and high levels of visual complexity, suggesting a pervasive imbalance in competitive interactions throughout the visual system of individuals with ASC.

Slower rate of binocular rivalry in autism.

An imbalance between cortical excitation and inhibition is a central component of many models of autistic neurobiology. We tested a potential behavioral footprint of this proposed imbalance using binocular rivalry, a visual phenomenon in which perceptual experience is thought to mirror the push and pull of excitatory and inhibitory cortical dynamics. In binocular rivalry, two monocularly presented images compete, leading to a percept that alternates between them. In a series of trials, we presented separate images of objects (e.g., a baseball and a broccoli) to each eye using a mirror stereoscope and asked human participants with autism and matched control subjects to continuously report which object they perceived, or whether they perceived a mixed percept. Individuals with autism demonstrated a slower rate of binocular rivalry alternations than matched control subjects, with longer durations of mixed percepts and an increased likelihood to revert to the previously perceived object when exiting a mixed percept. Critically, each of these findings was highly predictive of clinical measures of autistic symptomatology. Control "playback" experiments demonstrated that differences in neither response latencies nor response criteria could account for the atypical dynamics of binocular rivalry we observed in autistic spectrum conditions. Overall, these results may provide an index of atypical cortical dynamics that may underlie both the social and nonsocial symptoms of autism.

Tunnel vision: sharper gradient of spatial attention in autism.

Enhanced perception of detail has long been regarded a hallmark of autism spectrum conditions (ASC), but its origins are unknown. Normal sensitivity on all fundamental perceptual measures-visual acuity, contrast discrimination, and flicker detection-is strongly established in the literature. If individuals with ASC do not have superior low-level vision, how is perception of detail enhanced? We argue that this apparent paradox can be resolved by considering visual attention, which is known to enhance basic visual sensitivity, resulting in greater acuity and lower contrast thresholds. Here, we demonstrate that the focus of attention and concomitant enhancement of perception are sharper in human individuals with ASC than in matched controls. Using a simple visual acuity task embedded in a standard cueing paradigm, we mapped the spatial and temporal gradients of attentional enhancement by varying the distance and onset time of visual targets relative to an exogenous cue, which obligatorily captures attention. Individuals with ASC demonstrated a greater fall-off in performance with distance from the cue than controls, indicating a sharper spatial gradient of attention. Further, this sharpness was highly correlated with the severity of autistic symptoms in ASC, as well as autistic traits across both ASC and control groups. These findings establish the presence of a form of "tunnel vision" in ASC, with far-reaching implications for our understanding of the social and neurobiological aspects of autism.