Assistive AI in Lung Cancer Screening: A Retrospective Multinational Study in the United States and Japan.

Purpose To evaluate the impact of an artificial intelligence (AI) assistant for lung cancer screening on multinational clinical workflows. Materials and Methods An AI assistant for lung cancer screening was evaluated on two retrospective randomized multireader multicase studies where 627 (141 cancer-positive cases) low-dose chest CT cases were each read twice (with and without AI assistance) by experienced thoracic radiologists (six U.S.-based or six Japan-based radiologists), resulting in a total of 7524 interpretations. Positive cases were defined as those within 2 years before a pathology-confirmed lung cancer diagnosis. Negative cases were defined as those without any subsequent cancer diagnosis for at least 2 years and were enriched for a spectrum of diverse nodules. The studies measured the readers' level of suspicion (on a 0-100 scale), country-specific screening system scoring categories, and management recommendations. Evaluation metrics included the area under the receiver operating characteristic curve (AUC) for level of suspicion and sensitivity and specificity of recall recommendations. Results With AI assistance, the radiologists' AUC increased by 0.023 (0.70 to 0.72; P = .02) for the U.S. study and by 0.023 (0.93 to 0.96; P = .18) for the Japan study. Scoring system specificity for actionable findings increased 5.5% (57% to 63%; P < .001) for the U.S. study and 6.7% (23% to 30%; P < .001) for the Japan study. There was no evidence of a difference in corresponding sensitivity between unassisted and AI-assisted reads for the U.S. (67.3% to 67.5%; P = .88) and Japan (98% to 100%; P > .99) studies. Corresponding stand-alone AI AUC system performance was 0.75 (95% CI: 0.70, 0.81) and 0.88 (95% CI: 0.78, 0.97) for the U.S.- and Japan-based datasets, respectively. Conclusion The concurrent AI interface improved lung cancer screening specificity in both U.S.- and Japan-based reader studies, meriting further study in additional international screening environments. Keywords: Assistive Artificial Intelligence, Lung Cancer Screening, CT Supplemental material is available for this article. Published under a CC BY 4.0 license.

Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians.

Predictive artificial intelligence (AI) systems based on deep learning have been shown to achieve expert-level identification of diseases in multiple medical imaging settings, but can make errors in cases accurately diagnosed by clinicians and vice versa. We developed Complementarity-Driven Deferral to Clinical Workflow (CoDoC), a system that can learn to decide between the opinion of a predictive AI model and a clinical workflow. CoDoC enhances accuracy relative to clinician-only or AI-only baselines in clinical workflows that screen for breast cancer or tuberculosis (TB). For breast cancer screening, compared to double reading with arbitration in a screening program in the UK, CoDoC reduced false positives by 25% at the same false-negative rate, while achieving a 66% reduction in clinician workload. For TB triaging, compared to standalone AI and clinical workflows, CoDoC achieved a 5-15% reduction in false positives at the same false-negative rate for three of five commercially available predictive AI systems. To facilitate the deployment of CoDoC in novel futuristic clinical settings, we present results showing that CoDoC's performance gains are sustained across several axes of variation (imaging modality, clinical setting and predictive AI system) and discuss the limitations of our evaluation and where further validation would be needed. We provide an open-source implementation to encourage further research and application.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Deep Learning Detection of Active Pulmonary Tuberculosis at Chest Radiography Matched the Clinical Performance of Radiologists.

Background The World Health Organization (WHO) recommends chest radiography to facilitate tuberculosis (TB) screening. However, chest radiograph interpretation expertise remains limited in many regions. Purpose To develop a deep learning system (DLS) to detect active pulmonary TB on chest radiographs and compare its performance to that of radiologists. Materials and Methods A DLS was trained and tested using retrospective chest radiographs (acquired between 1996 and 2020) from 10 countries. To improve generalization, large-scale chest radiograph pretraining, attention pooling, and semisupervised learning ("noisy-student") were incorporated. The DLS was evaluated in a four-country test set (China, India, the United States, and Zambia) and in a mining population in South Africa, with positive TB confirmed with microbiological tests or nucleic acid amplification testing (NAAT). The performance of the DLS was compared with that of 14 radiologists. The authors studied the efficacy of the DLS compared with that of nine radiologists using the Obuchowski-Rockette-Hillis procedure. Given WHO targets of 90% sensitivity and 70% specificity, the operating point of the DLS (0.45) was prespecified to favor sensitivity. Results A total of 165 754 images in 22 284 subjects (mean age, 45 years; 21% female) were used for model development and testing. In the four-country test set (1236 subjects, 17% with active TB), the receiver operating characteristic (ROC) curve of the DLS was higher than those for all nine India-based radiologists, with an area under the ROC curve of 0.89 (95% CI: 0.87, 0.91). Compared with these radiologists, at the prespecified operating point, the DLS sensitivity was higher (88% vs 75%, P < .001) and specificity was noninferior (79% vs 84%, P = .004). Trends were similar within other patient subgroups, in the South Africa data set, and across various TB-specific chest radiograph findings. In simulations, the use of the DLS to identify likely TB-positive chest radiographs for NAAT confirmation reduced the cost by 40%-80% per TB-positive patient detected. Conclusion A deep learning method was found to be noninferior to radiologists for the determination of active tuberculosis on digital chest radiographs. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by van Ginneken in this issue.

A mobile-optimized artificial intelligence system for gestational age and fetal malpresentation assessment.

Background: Fetal ultrasound is an important component of antenatal care, but shortage of adequately trained healthcare workers has limited its adoption in low-to-middle-income countries. This study investigated the use of artificial intelligence for fetal ultrasound in under-resourced settings. Methods: Blind sweep ultrasounds, consisting of six freehand ultrasound sweeps, were collected by sonographers in the USA and Zambia, and novice operators in Zambia. We developed artificial intelligence (AI) models that used blind sweeps to predict gestational age (GA) and fetal malpresentation. AI GA estimates and standard fetal biometry estimates were compared to a previously established ground truth, and evaluated for difference in absolute error. Fetal malpresentation (non-cephalic vs cephalic) was compared to sonographer assessment. On-device AI model run-times were benchmarked on Android mobile phones. Results: Here we show that GA estimation accuracy of the AI model is non-inferior to standard fetal biometry estimates (error difference -1.4 ± 4.5 days, 95% CI -1.8, -0.9, n = 406). Non-inferiority is maintained when blind sweeps are acquired by novice operators performing only two of six sweep motion types. Fetal malpresentation AUC-ROC is 0.977 (95% CI, 0.949, 1.00, n = 613), sonographers and novices have similar AUC-ROC. Software run-times on mobile phones for both diagnostic models are less than 3 s after completion of a sweep. Conclusions: The gestational age model is non-inferior to the clinical standard and the fetal malpresentation model has high AUC-ROCs across operators and devices. Our AI models are able to run on-device, without internet connectivity, and provide feedback scores to assist in upleveling the capabilities of lightly trained ultrasound operators in low resource settings.

Deep Learning to Detect OCT-derived Diabetic Macular Edema from Color Retinal Photographs: A Multicenter Validation Study.

PURPOSE: To validate the generalizability of a deep learning system (DLS) that detects diabetic macular edema (DME) from 2-dimensional color fundus photographs (CFP), for which the reference standard for retinal thickness and fluid presence is derived from 3-dimensional OCT. DESIGN: Retrospective validation of a DLS across international datasets. PARTICIPANTS: Paired CFP and OCT of patients from diabetic retinopathy (DR) screening programs or retina clinics. The DLS was developed using data sets from Thailand, the United Kingdom, and the United States and validated using 3060 unique eyes from 1582 patients across screening populations in Australia, India, and Thailand. The DLS was separately validated in 698 eyes from 537 screened patients in the United Kingdom with mild DR and suspicion of DME based on CFP. METHODS: The DLS was trained using DME labels from OCT. The presence of DME was based on retinal thickening or intraretinal fluid. The DLS's performance was compared with expert grades of maculopathy and to a previous proof-of-concept version of the DLS. We further simulated the integration of the current DLS into an algorithm trained to detect DR from CFP. MAIN OUTCOME MEASURES: The superiority of specificity and noninferiority of sensitivity of the DLS for the detection of center-involving DME, using device-specific thresholds, compared with experts. RESULTS: The primary analysis in a combined data set spanning Australia, India, and Thailand showed the DLS had 80% specificity and 81% sensitivity, compared with expert graders, who had 59% specificity and 70% sensitivity. Relative to human experts, the DLS had significantly higher specificity (P = 0.008) and noninferior sensitivity (P < 0.001). In the data set from the United Kingdom, the DLS had a specificity of 80% (P < 0.001 for specificity of >50%) and a sensitivity of 100% (P = 0.02 for sensitivity of > 90%). CONCLUSIONS: The DLS can generalize to multiple international populations with an accuracy exceeding that of experts. The clinical value of this DLS to reduce false-positive referrals, thus decreasing the burden on specialist eye care, warrants a prospective evaluation.

International evaluation of an AI system for breast cancer screening.

Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful1. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives2. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. To assess its performance in the clinical setting, we curated a large representative dataset from the UK and a large enriched dataset from the USA. We show an absolute reduction of 5.7% and 1.2% (USA and UK) in false positives and 9.4% and 2.7% in false negatives. We provide evidence of the ability of the system to generalize from the UK to the USA. In an independent study of six radiologists, the AI system outperformed all of the human readers: the area under the receiver operating characteristic curve (AUC-ROC) for the AI system was greater than the AUC-ROC for the average radiologist by an absolute margin of 11.5%. We ran a simulation in which the AI system participated in the double-reading process that is used in the UK, and found that the AI system maintained non-inferior performance and reduced the workload of the second reader by 88%. This robust assessment of the AI system paves the way for clinical trials to improve the accuracy and efficiency of breast cancer screening.

Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation.

BackgroundDeep learning has the potential to augment the use of chest radiography in clinical radiology, but challenges include poor generalizability, spectrum bias, and difficulty comparing across studies.PurposeTo develop and evaluate deep learning models for chest radiograph interpretation by using radiologist-adjudicated reference standards.Materials and MethodsDeep learning models were developed to detect four findings (pneumothorax, opacity, nodule or mass, and fracture) on frontal chest radiographs. This retrospective study used two data sets. Data set 1 (DS1) consisted of 759 611 images from a multicity hospital network and ChestX-ray14 is a publicly available data set with 112 120 images. Natural language processing and expert review of a subset of images provided labels for 657 954 training images. Test sets consisted of 1818 and 1962 images from DS1 and ChestX-ray14, respectively. Reference standards were defined by radiologist-adjudicated image review. Performance was evaluated by area under the receiver operating characteristic curve analysis, sensitivity, specificity, and positive predictive value. Four radiologists reviewed test set images for performance comparison. Inverse probability weighting was applied to DS1 to account for positive radiograph enrichment and estimate population-level performance.ResultsIn DS1, population-adjusted areas under the receiver operating characteristic curve for pneumothorax, nodule or mass, airspace opacity, and fracture were, respectively, 0.95 (95% confidence interval [CI]: 0.91, 0.99), 0.72 (95% CI: 0.66, 0.77), 0.91 (95% CI: 0.88, 0.93), and 0.86 (95% CI: 0.79, 0.92). With ChestX-ray14, areas under the receiver operating characteristic curve were 0.94 (95% CI: 0.93, 0.96), 0.91 (95% CI: 0.89, 0.93), 0.94 (95% CI: 0.93, 0.95), and 0.81 (95% CI: 0.75, 0.86), respectively.ConclusionExpert-level models for detecting clinically relevant chest radiograph findings were developed for this study by using adjudicated reference standards and with population-level performance estimation. Radiologist-adjudicated labels for 2412 ChestX-ray14 validation set images and 1962 test set images are provided.© RSNA, 2019Online supplemental material is available for this article.See also the editorial by Chang in this issue.

The subjective-objective mismatch in sleep perception among those with insomnia and sleep apnea.

The diagnosis and management of insomnia relies primarily on clinical history. However, patient self-report of sleep-wake times may not agree with objective measurements. We hypothesized that those with shallow or fragmented sleep would under-report sleep quantity, and that this might account for some of the mismatch. We compared objective and subjective sleep-wake times for 277 patients who underwent diagnostic polysomnography. The group included those with insomnia symptoms (n = 92), obstructive sleep apnea (n = 66) or both (n = 119). Mismatch of wake duration was context dependent: all three groups overestimated sleep latency but underestimated wakefulness after sleep onset. The insomnia group underestimated total sleep time by a median of 81 min. However, contrary to our hypothesis, measures of fragmentation (N1, arousal index, sleep efficiency, etc.) did not correlate with the subjective sleep duration estimates. To unmask a potential relationship between sleep architecture and subjective duration, we tested three hypotheses: N1 is perceived as wake; sleep bouts under 10 min are perceived as wake; or N1 and N2 are perceived in a weighted fashion. None of these hypotheses exposed a match between subjective and objective sleep duration. We show only modest performance of a Naïve Bayes Classifier algorithm for predicting mismatch using clinical and polysomnographic variables. Subjective-objective mismatch is common in patients reporting insomnia symptoms. We conclude that mismatch was not attributable to commonly measured polysomnographic measures of fragmentation. Further insight is needed into the complex relationships between subjective perception of sleep and conventional, objective measurements.

Temporal- and stimulus-based constraints to interactive activation during visual word recognition in adult readers.

In models of visual word recognition that incorporate an interactive activation framework, activation spreads from orthographic and phonological units to semantic units and from semantic units back to phonological and orthographic units. The present research examined whether semantic feedback changes over the time course of lexical processing and as a function of stimulus quality. Using a mediated priming paradigm, prime-target word pairs were associatively related (frog-toad), homophonically mediated (frog-towed), or orthographically mediated (frog-told). Evidence of semantic feedback to both orthography and phonology was found when the prime duration was 146 ms (Experiment 1) and only to phonology when the prime duration was 253 ms (Experiment 2a). However, when the prime duration was 253 ms and target words were degraded (Experiment 2b), feedback spread only to orthography. The results suggest that the dynamics of semantic feedback change as the function of processing demands in the visual word recognition system.

Sleep disruption due to hospital noises: a prospective evaluation.

BACKGROUND: Sleep plays a critical role in maintaining health and well-being; however, patients who are hospitalized are frequently exposed to noise that can disrupt sleep. Efforts to attenuate hospital noise have been limited by incomplete information on the interaction between sounds and sleep physiology. OBJECTIVE: To determine profiles of acoustic disruption of sleep by examining the cortical (encephalographic) arousal responses during sleep to typical hospital noises by sound level and type and sleep stage. DESIGN: 3-day polysomnographic study. SETTING: Sound-attenuated sleep laboratory. PARTICIPANTS: Volunteer sample of 12 healthy participants. INTERVENTION: Baseline (sham) night followed by 2 intervention nights with controlled presentation of 14 sounds that are common in hospitals (for example, voice, intravenous alarm, phone, ice machine, outside traffic, and helicopter). The sounds were administered at calibrated, increasing decibel levels (40 to 70 dBA [decibels, adjusted for the range of normal hearing]) during specific sleep stages. MEASUREMENTS: Encephalographic arousals, by using established criteria, during rapid eye movement (REM) sleep and non-REM (NREM) sleep stages 2 and 3. RESULTS: Sound presentations yielded arousal response curves that varied because of sound level and type and sleep stage. Electronic sounds were more arousing than other sounds, including human voices, and there were large differences in responses by sound type. As expected, sounds in NREM stage 3 were less likely to cause arousals than sounds in NREM stage 2; unexpectedly, the probability of arousal to sounds presented in REM sleep varied less by sound type than when presented in NREM sleep and caused a greater and more sustained elevation of instantaneous heart rate. LIMITATIONS: The study included only 12 participants. Results for these healthy persons may underestimate the effects of noise on sleep in patients who are hospitalized. CONCLUSION: Sounds during sleep influence both cortical brain activity and cardiovascular function. This study systematically quantifies the disruptive capacity of a range of hospital sounds on sleep, providing evidence that is essential to improving the acoustic environments of new and existing health care facilities to enable the highest quality of care. PRIMARY FUNDING SOURCE: Academy of Architecture for Health, Facilities Guidelines Institute, and The Center for Health Design.

DiBa: a data-driven Bayesian algorithm for sleep spindle detection.

Although the spontaneous brain rhythms of sleep have commanded much recent interest, their detection and analysis remains suboptimal. In this paper, we develop a data-driven Bayesian algorithm for sleep spindle detection on the electroencephalography (EEG). The algorithm exploits the Karhunen-Loève transform and Bayesian hypothesis testing to produce the instantaneous probability of a spindle's presence with maximal resolution. In addition to possessing flexibility, transparency, and scalability, this algorithm could perform at levels superior to standard methods for EEG event detection.

Covert waking brain activity reveals instantaneous sleep depth.

The neural correlates of the wake-sleep continuum remain incompletely understood, limiting the development of adaptive drug delivery systems for promoting sleep maintenance. The most useful measure for resolving early positions along this continuum is the alpha oscillation, an 8-13 Hz electroencephalographic rhythm prominent over posterior scalp locations. The brain activation signature of wakefulness, alpha expression discloses immediate levels of alertness and dissipates in concert with fading awareness as sleep begins. This brain activity pattern, however, is largely ignored once sleep begins. Here we show that the intensity of spectral power in the alpha band actually continues to disclose instantaneous responsiveness to noise--a measure of sleep depth--throughout a night of sleep. By systematically challenging sleep with realistic and varied acoustic disruption, we found that sleepers exhibited markedly greater sensitivity to sounds during moments of elevated alpha expression. This result demonstrates that alpha power is not a binary marker of the transition between sleep and wakefulness, but carries rich information about immediate sleep stability. Further, it shows that an empirical and ecologically relevant form of sleep depth is revealed in real-time by EEG spectral content in the alpha band, a measure that affords prediction on the order of minutes. This signal, which transcends the boundaries of classical sleep stages, could potentially be used for real-time feedback to novel, adaptive drug delivery systems for inducing sleep.

Spontaneous brain rhythms predict sleep stability in the face of noise.

Quality sleep is an essential part of health and well-being. Yet fractured sleep is disturbingly prevalent in our society, partly due to insults from a variety of noises [1]. Common experience suggests that this fragility of sleep is highly variable between people, but it is unclear what mechanisms drive these differences. Here we show that it is possible to predict an individual's ability to maintain sleep in the face of sound using spontaneous brain rhythms from electroencephalography (EEG). The sleep spindle is a thalamocortical rhythm manifested on the EEG as a brief 11-15 Hz oscillation and is thought to be capable of modulating the influence of external stimuli [2]. Its rate of occurrence, while variable across people, is stable across nights [3]. We found that individuals who generated more sleep spindles during a quiet night of sleep went on to exhibit higher tolerance for noise during a subsequent, noisy night of sleep. This result shows that the sleeping brain's spontaneous activity heralds individual resilience to disruptive stimuli. Our finding sets the stage for future studies that attempt to augment spindle production to enhance sleep continuity when confronted with noise.