Are better AI algorithms for breast cancer detection also better at predicting risk? A paired case-control study.

BACKGROUND: There is increasing evidence that artificial intelligence (AI) breast cancer risk evaluation tools using digital mammograms are highly informative for 1-6 years following a negative screening examination. We hypothesized that algorithms that have previously been shown to work well for cancer detection will also work well for risk assessment and that performance of algorithms for detection and risk assessment is correlated. METHODS: To evaluate our hypothesis, we designed a case-control study using paired mammograms at diagnosis and at the previous screening visit. The study included n = 3386 women from the OPTIMAM registry, that includes mammograms from women diagnosed with breast cancer in the English breast screening program 2010-2019. Cases were diagnosed with invasive breast cancer or ductal carcinoma in situ at screening and were selected if they had a mammogram available at the screening examination that led to detection, and a paired mammogram at their previous screening visit 3y prior to detection when no cancer was detected. Controls without cancer were matched 1:1 to cases based on age (year), screening site, and mammography machine type. Risk assessment was conducted using a deep-learning model designed for breast cancer risk assessment (Mirai), and three open-source deep-learning algorithms designed for breast cancer detection. Discrimination was assessed using a matched area under the curve (AUC) statistic. RESULTS: Overall performance using the paired mammograms followed the same order by algorithm for risk assessment (AUC range 0.59-0.67) and detection (AUC 0.81-0.89), with Mirai performing best for both. There was also a correlation in performance for risk and detection within algorithms by cancer size, with much greater accuracy for large cancers (30 mm+, detection AUC: 0.88-0.92; risk AUC: 0.64-0.74) than smaller cancers (0 to < 10 mm, detection AUC: 0.73-0.86, risk AUC: 0.54-0.64). Mirai was relatively strong for risk assessment of smaller cancers (0 to < 10 mm, risk, Mirai AUC: 0.64 (95% CI 0.57 to 0.70); other algorithms AUC 0.54-0.56). CONCLUSIONS: Improvements in risk assessment could stem from enhancing cancer detection capabilities of smaller cancers. Other state-of-the-art AI detection algorithms with high performance for smaller cancers might achieve relatively high performance for risk assessment.

Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective, multicentre study.

BACKGROUND: Artificial intelligence (AI) systems for automated chest x-ray interpretation hold promise for standardising reporting and reducing delays in health systems with shortages of trained radiologists. Yet, there are few freely accessible AI systems trained on large datasets for practitioners to use with their own data with a view to accelerating clinical deployment of AI systems in radiology. We aimed to contribute an AI system for comprehensive chest x-ray abnormality detection. METHODS: In this retrospective cohort study, we developed open-source neural networks, X-Raydar and X-Raydar-NLP, for classifying common chest x-ray findings from images and their free-text reports. Our networks were developed using data from six UK hospitals from three National Health Service (NHS) Trusts (University Hospitals Coventry and Warwickshire NHS Trust, University Hospitals Birmingham NHS Foundation Trust, and University Hospitals Leicester NHS Trust) collectively contributing 2 513 546 chest x-ray studies taken from a 13-year period (2006-19), which yielded 1 940 508 usable free-text radiological reports written by the contemporary assessing radiologist (collectively referred to as the "historic reporters") and 1 896 034 frontal images. Chest x-rays were labelled using a taxonomy of 37 findings by a custom-trained natural language processing (NLP) algorithm, X-Raydar-NLP, from the original free-text reports. X-Raydar-NLP was trained on 23 230 manually annotated reports and tested on 4551 reports from all hospitals. 1 694 921 labelled images from the training set and 89 238 from the validation set were then used to train a multi-label image classifier. Our algorithms were evaluated on three retrospective datasets: a set of exams sampled randomly from the full NHS dataset reported during clinical practice and annotated using NLP (n=103 328); a consensus set sampled from all six hospitals annotated by three expert radiologists (two independent annotators for each image and a third consultant to facilitate disagreement resolution) under research conditions (n=1427); and an independent dataset, MIMIC-CXR, consisting of NLP-annotated exams (n=252 374). FINDINGS: X-Raydar achieved a mean AUC of 0·919 (SD 0·039) on the auto-labelled set, 0·864 (0·102) on the consensus set, and 0·842 (0·074) on the MIMIC-CXR test, demonstrating similar performance to the historic clinical radiologist reporters, as assessed on the consensus set, for multiple clinically important findings, including pneumothorax, parenchymal opacification, and parenchymal mass or nodules. On the consensus set, X-Raydar outperformed historical reporter balanced accuracy with significance on 27 of 37 findings, was non-inferior on nine, and inferior on one finding, resulting in an average improvement of 13·3% (SD 13·1) to 0·763 (0·110), including a mean 5·6% (13·2) improvement in critical findings to 0·826 (0·119). INTERPRETATION: Our study shows that automated classification of chest x-rays under a comprehensive taxonomy can achieve performance levels similar to those of historical reporters and exhibit robust generalisation to external data. The open-sourced neural networks can serve as foundation models for further research and are freely available to the research community. FUNDING: Wellcome Trust.

An optimization framework to guide the choice of thresholds for risk-based cancer screening.

It is uncommon for risk groups defined by statistical or artificial intelligence (AI) models to be chosen by jointly considering model performance and potential interventions available. We develop a framework to rapidly guide choice of risk groups in this manner, and apply it to guide breast cancer screening intervals using an AI model. Linear programming is used to define risk groups that minimize expected advanced cancer incidence subject to resource constraints. In the application risk stratification performance is estimated from a case-control study (2044 cases, 1:1 matching), and other parameters are taken from screening trials and the screening programme in England. Under the model, re-screening in 1 year for the highest 4% AI model risk, in 3 years for the middle 64%, and in 4 years for 32% of the population at lowest risk, was expected to reduce the number of advanced cancers diagnosed by approximately 18 advanced cancers per 1000 diagnosed with triennial screening, for the same average number of screens in the population as triennial screening for all. Sensitivity analyses found the choice of thresholds was robust to model parameters, but the estimated reduction in advanced cancers was not precise and requires further evaluation. Our framework helps define thresholds with the greatest chance of success for reducing the population health burden of cancer when used in risk-adapted screening, which should be further evaluated such as in health-economic modelling based on computer simulation models, and real-world evaluations.