Differences and similarities in false interpretations by AI CAD and radiologists in screening mammography.

OBJECTIVE: We aimed to evaluate the false interpretations between artificial intelligence (AI) and radiologists in screening mammography to get a better understanding of how the distribution of diagnostic mistakes might change when moving from entirely radiologist-driven to AI-integrated breast cancer screening. METHODS AND MATERIALS: This retrospective case-control study was based on a mammography screening cohort from 2008 to 2015. The final study population included screening examinations for 714 women diagnosed with breast cancer and 8029 randomly selected healthy controls. Oversampling of controls was applied to attain a similar cancer proportion as in the source screening cohort. We examined how false-positive (FP) and false-negative (FN) assessments by AI, the first reader (RAD 1) and the second reader (RAD 2), were associated with age, density, tumor histology and cancer invasiveness in a single- and double-reader setting. RESULTS: For each reader, the FN assessments were distributed between low- and high-density females with 53 (42%) and 72 (58%) for AI; 59 (36%) and 104 (64%) for RAD 1 and 47 (36%) and 84 (64%) for RAD 2. The corresponding numbers for FP assessments were 1820 (47%) and 2016 (53%) for AI; 1568 (46%) and 1834 (54%) for RAD 1 and 1190 (43%) and 1610 (58%) for RAD 2. For ductal cancer, the FN assessments were 79 (77%) for AI CAD; with 120 (83%) for RAD 1 and with 96 (16%) for RAD 2. For the double-reading simulation, the FP assessments were distributed between younger and older females with 2828 (2.5%) and 1554 (1.4%) for RAD 1 + RAD 2; 3850 (3.4%) and 2940 (2.6%) for AI+RAD 1 and 3430 (3%) and 2772 (2.5%) for AI+RAD 2. CONCLUSION: The most pronounced decrease in FN assessments was noted for females over the age of 55 and for high density-women. In conclusion, AI could have an important complementary role when combined with radiologists to increase sensitivity for high-density and older females. ADVANCES IN KNOWLEDGE: Our results highlight the potential impact of integrating AI in breast cancer screening, particularly to improve interpretation accuracy. The use of AI could enhance screening outcomes for high-density and older females.

Implications for downstream workload based on calibrating an artificial intelligence detection algorithm by standalone-reader or combined-reader sensitivity matching.

Purpose: In double-reading of screening mammograms, artificial intelligence (AI) algorithms hold promise as a potential replacement for one of the two readers. The choice of operating point, or abnormality threshold, for the AI algorithm will affect cancer detection and workload. In our retrospective study, the baseline approach was based on matching stand-alone reader sensitivity, while the alternative approach was based on matching the combined-reader sensitivity of two humans and of AI plus human. Approach: Full-field digital screening mammograms within the Stockholm County area between February 1, 2012, and December 30, 2015, acquired on Philips equipment, were collected. All exams of women with breast cancer within 23 months of screening and a random selection of healthy controls were included. An exam-level continuous AI abnormality score was generated (Insight MMG from Lunit Inc). Sensitivity and abnormal interpretation rates were estimated for operating points defined by the standalone-reader approach and the combined-reader approach. Results: The study population included 1684 exams of women with breast cancer and 5024 exams of healthy women. Observations of healthy women were up-sampled to attain a realistic proportion of cancer. The observed sensitivity for reader 1, 2 and 1+2 was 69.7%, 75.6%, and 78.6%, respectively, at an abnormal interpretation rate of 4.4%, 4.6%, and 6.1%, respectively. For the combination of reader 1 + AI we estimated a sensitivity of 82.4% for standalone-reader matching and 78.6% for combined-reader matching, at an abnormal interpretation rate of 12.6% and 7.0%, respectively. Conclusions: Setting the operating point by matching stand-alone AI stand-alone with a radiologist will nearly double the downstream workload compared to a modest increase of 15% for the alternative method of matching sensitivity between the combination of AI and a radiologist with two radiologists.

Effect of artificial intelligence-based triaging of breast cancer screening mammograms on cancer detection and radiologist workload: a retrospective simulation study.

BACKGROUND: We examined the potential change in cancer detection when using an artificial intelligence (AI) cancer-detection software to triage certain screening examinations into a no radiologist work stream, and then after regular radiologist assessment of the remainder, triage certain screening examinations into an enhanced assessment work stream. The purpose of enhanced assessment was to simulate selection of women for more sensitive screening promoting early detection of cancers that would otherwise be diagnosed as interval cancers or as next-round screen-detected cancers. The aim of the study was to examine how AI could reduce radiologist workload and increase cancer detection. METHODS: In this retrospective simulation study, all women diagnosed with breast cancer who attended two consecutive screening rounds were included. Healthy women were randomly sampled from the same cohort; their observations were given elevated weight to mimic a frequency of 0·7% incident cancer per screening interval. Based on the prediction score from a commercially available AI cancer detector, various cutoff points for the decision to channel women to the two new work streams were examined in terms of missed and additionally detected cancer. FINDINGS: 7364 women were included in the study sample: 547 were diagnosed with breast cancer and 6817 were healthy controls. When including 60%, 70%, or 80% of women with the lowest AI scores in the no radiologist stream, the proportion of screen-detected cancers that would have been missed were 0, 0·3% (95% CI 0·0-4·3), or 2·6% (1·1-5·4), respectively. When including 1% or 5% of women with the highest AI scores in the enhanced assessment stream, the potential additional cancer detection was 24 (12%) or 53 (27%) of 200 subsequent interval cancers, respectively, and 48 (14%) or 121 (35%) of 347 next-round screen-detected cancers, respectively. INTERPRETATION: Using a commercial AI cancer detector to triage mammograms into no radiologist assessment and enhanced assessment could potentially reduce radiologist workload by more than half, and pre-emptively detect a substantial proportion of cancers otherwise diagnosed later. FUNDING: Stockholm City Council.

External Evaluation of 3 Commercial Artificial Intelligence Algorithms for Independent Assessment of Screening Mammograms.

Importance: A computer algorithm that performs at or above the level of radiologists in mammography screening assessment could improve the effectiveness of breast cancer screening. Objective: To perform an external evaluation of 3 commercially available artificial intelligence (AI) computer-aided detection algorithms as independent mammography readers and to assess the screening performance when combined with radiologists. Design, Setting, and Participants: This retrospective case-control study was based on a double-reader population-based mammography screening cohort of women screened at an academic hospital in Stockholm, Sweden, from 2008 to 2015. The study included 8805 women aged 40 to 74 years who underwent mammography screening and who did not have implants or prior breast cancer. The study sample included 739 women who were diagnosed as having breast cancer (positive) and a random sample of 8066 healthy controls (negative for breast cancer). Main Outcomes and Measures: Positive follow-up findings were determined by pathology-verified diagnosis at screening or within 12 months thereafter. Negative follow-up findings were determined by a 2-year cancer-free follow-up. Three AI computer-aided detection algorithms (AI-1, AI-2, and AI-3), sourced from different vendors, yielded a continuous score for the suspicion of cancer in each mammography examination. For a decision of normal or abnormal, the cut point was defined by the mean specificity of the first-reader radiologists (96.6%). Results: The median age of study participants was 60 years (interquartile range, 50-66 years) for 739 women who received a diagnosis of breast cancer and 54 years (interquartile range, 47-63 years) for 8066 healthy controls. The cases positive for cancer comprised 618 (84%) screen detected and 121 (16%) clinically detected within 12 months of the screening examination. The area under the receiver operating curve for cancer detection was 0.956 (95% CI, 0.948-0.965) for AI-1, 0.922 (95% CI, 0.910-0.934) for AI-2, and 0.920 (95% CI, 0.909-0.931) for AI-3. At the specificity of the radiologists, the sensitivities were 81.9% for AI-1, 67.0% for AI-2, 67.4% for AI-3, 77.4% for first-reader radiologist, and 80.1% for second-reader radiologist. Combining AI-1 with first-reader radiologists achieved 88.6% sensitivity at 93.0% specificity (abnormal defined by either of the 2 making an abnormal assessment). No other examined combination of AI algorithms and radiologists surpassed this sensitivity level. Conclusions and Relevance: To our knowledge, this study is the first independent evaluation of several AI computer-aided detection algorithms for screening mammography. The results of this study indicated that a commercially available AI computer-aided detection algorithm can assess screening mammograms with a sufficient diagnostic performance to be further evaluated as an independent reader in prospective clinical trials. Combining the first readers with the best algorithm identified more cases positive for cancer than combining the first readers with second readers.

Range of Radiologist Performance in a Population-based Screening Cohort of 1 Million Digital Mammography Examinations.

Background There is great interest in developing artificial intelligence (AI)-based computer-aided detection (CAD) systems for use in screening mammography. Comparative performance benchmarks from true screening cohorts are needed. Purpose To determine the range of human first-reader performance measures within a population-based screening cohort of 1 million screening mammograms to gauge the performance of emerging AI CAD systems. Materials and Methods This retrospective study consisted of all screening mammograms in women aged 40-74 years in Stockholm County, Sweden, who underwent screening with full-field digital mammography between 2008 and 2015. There were 110 interpreting radiologists, of whom 24 were defined as high-volume readers (ie, those who interpreted more than 5000 annual screening mammograms). A true-positive finding was defined as the presence of a pathology-confirmed cancer within 12 months. Performance benchmarks included sensitivity and specificity, examined per quartile of radiologists' performance. First-reader sensitivity was determined for each tumor subgroup, overall and by quartile of high-volume reader sensitivity. Screening outcomes were examined based on the first reader's sensitivity quartile with 10 000 screening mammograms per quartile. Linear regression models were fitted to test for a linear trend across quartiles of performance. Results A total of 418 041 women (mean age, 54 years ± 10 [standard deviation]) were included, and 1 186 045 digital mammograms were evaluated, with 972 899 assessed by high-volume readers. Overall sensitivity was 73% (95% confidence interval [CI]: 69%, 77%), and overall specificity was 96% (95% CI: 95%, 97%). The mean values per quartile of high-volume reader performance ranged from 63% to 84% for sensitivity and from 95% to 98% for specificity. The sensitivity difference was very large for basal cancers, with the least sensitive and most sensitive high-volume readers detecting 53% and 89% of cancers, respectively (P < .001). Conclusion Benchmarks showed a wide range of performance differences between high-volume readers. Sensitivity varied by tumor characteristics. © RSNA, 2020 Online supplemental material is available for this article.