Impact of and interaction between the availability of prior examinations and DBT on the interpretation of negative and benign mammograms.

RATIONALE AND OBJECTIVES: To assess the interaction between the availability of prior examinations and digital breast tomosynthesis (DBT) in decisions to recall a woman during interpretation of mammograms. MATERIALS AND METHODS: Eight radiologists independently interpreted twice 36 mammography examinations, each of which had current and prior full-field digital mammography images (FFDM) and DBT under a Health Insurance Portability and Accountability Act-compliant, institutional review board-approved protocol (written consent waived). During the first reading, three sequential ratings were provided using FFDM only, followed by FFDM + DBT, and then followed by FFDM + DBT + priors. The second reading included FFDM only, then FFDM + priors, and then FFDM + priors + DBT. Twenty-two benign cases clinically recalled, 12 negative/benign examinations (not recalled), and two verified cancer cases were included. Recall recommendations and interaction between the effect of priors and DBT on decisions were assessed (P = .05 significance level) using generalized linear model (PROC GLIMMIX, SAS, version 9.3; SAS Institute, Cary, NC) accounting for case and reader variability. RESULTS: Average recall rates in noncancer cases were significantly reduced (51%; P < .001) with the addition of DBT and with addition of priors (23%; P = .01). In absolute terms, the addition of DBT to FFDM reduced the recall rates from 0.67 to 0.42 and from 0.54 to 0.27 when DBT was available before and after priors, respectively. Recall reductions were from 0.64 to 0.54 and from 0.42 to 0.33 when priors were available before and after DBT, respectively. Regardless of the sequence in presentation, there were no statistically significant interactions between the effect of availability of DBT and priors (P = .80). CONCLUSIONS: Availability of both priors and DBT are independent primary factors in reducing recall recommendations during mammographic interpretations.

Detection and classification of calcifications on digital breast tomosynthesis and 2D digital mammography: a comparison.

OBJECTIVE: The purpose of this article is to compare the ability of digital breast tomosynthesis and full field digital mammography (FFDM) to detect and characterize calcifications. MATERIALS AND METHODS: One hundred paired examinations were performed utilizing FFDM and digital breast tomosynthesis. Twenty biopsy-proven cancers, 40 biopsy-proven benign calcifications, and 40 randomly selected negative screening studies were retrospectively reviewed by five radiologists in a crossed multireader multimodal observer performance study. Data collected included the presence of calcifications and forced BI-RADS scores. Receiver operator curve analysis using BI-RADS was performed. RESULTS: Overall calcification detection sensitivity was higher for FFDM (84% [95% CI, 79-88%]) than for digital breast tomosynthesis (75% [95% CI, 70-80%]). [corrected] In the cancer cohort, 75 (76%) of 99 interpretations identified calcification in both modes. Of those, a BI-RADS score less than or equal to 2 was rendered in three (4%) and nine (12%) cases with FFDM and digital breast tomosynthesis, respectively. In the benign cohort, 123 (62%) of 200 interpretations identified calcifications in both modes. Of those, a BI-RADS score greater than or equal to 3 was assigned in 105 (85%) and 93 (76%) cases with FFDM and digital breast tomosynthesis, respectively. There was no significant difference in the nonparametric computed area under the receiver operating characteristic curves (AUC) using the BI-RADS scores (FFDM, AUC = 0.76 and SD = 0.03; digital breast tomosynthesis, AUC = 0.72 and SD = 0.04 [p = 0.1277]). CONCLUSION: In this small data set, FFDM appears to be slightly more sensitive than digital breast tomosynthesis for the detection of calcification. However, diagnostic performance as measured by area under the curve using BI-RADS was not significantly different. With improvements in processing algorithms and display, digital breast tomosynthesis could potentially be improved for this purpose.

Binary and multi-category ratings in a laboratory observer performance study: a comparison.

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

Agreement of the order of overall performance levels under different reading paradigms.

RATIONALE AND OBJECTIVES: To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms. MATERIALS AND METHODS: We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations. RESULTS: All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733). CONCLUSION: The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations.

PURPOSE: To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study. MATERIALS AND METHODS: This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates. RESULTS: On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01). CONCLUSION: Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well.

Trends in recall, biopsy, and positive biopsy rates for screening mammography in an academic practice.

PURPOSE: To retrospectively evaluate whether recall, biopsy, and positive biopsy rates for a group of radiologists who met requirements of Mammography Quality Standards Act of 1992 (MQSA) demonstrated any change over time during a 27-month period (nine consecutive calendar quarters). MATERIALS AND METHODS: Institutional review board approved study protocol, and informed consent was waived. All screening mammograms that had been interpreted by MQSA-qualified radiologists between January 1, 2001, and March 31, 2003, were reviewed. Group recall rates, biopsy rates, and detected cancer rates for nine calendar quarters were computed and attributed to performance date of original screening mammogram. Type of biopsy performed was classified as follows: stereotactic vacuum-assisted biopsy, ultrasonography (US)-guided core biopsy, US-guided fine-needle aspiration biopsy, surgical excision, and multiple biopsies. chi(2) Test for trend (two sided) and linear regression were used to assess trends over time for recall and biopsy rates, biopsy rates according to type of biopsy performed, and percentage of biopsy results positive for cancer. RESULTS: Group recall rate did not show a statistically significant trend during period studied (P = .59). Biopsy rates increased significantly from 13.02 to 20.12 per 1000 screening examinations (P < .001). A corresponding substantial decrease was seen in percentage of biopsies in which malignancy was found, although this trend was not statistically significant (P = .24). A significant increase (from 4.72 to 9.88 per 1000 screening examinations) was found in rate of stereotactic vacuum-assisted 11-gauge core biopsies performed (P < .001). CONCLUSION: Observed increase in biopsy rates reinforces the need to carefully select patients for biopsy to achieve efficient, efficacious, and cost-effective programs for early detection of breast cancers.

Recall and detection rates in screening mammography.

BACKGROUND: The authors investigated the correlation between recall and detection rates in a group of 10 radiologists who had read a high volume of screening mammograms in an academic institution. METHODS: Practice-related and outcome-related databases of verified cases were used to compute recall rates and tumor detection rates for a group of 10 Mammography Quality Standard Act (MQSA)-certified radiologists who interpreted a total of 98,668 screening mammograms during the years 2000, 2001, and 2002. The relation between recall and detection rates for these individuals was investigated using parametric Pearson (r) and nonparametric Spearman (rho) correlation coefficients. The effect of the volume of mammograms interpreted by individual radiologists was assessed using partial correlations controlling for total reading volumes. RESULTS: A wide variability of recall rates (range, 7.7-17.2%) and detection rates (range, 2.6-5.4 per 1000 mammograms) was observed in the current study. A statistically significant correlation (P < 0.05) between recall and detection rates was observed in this group of 10 experienced radiologists. The results remained significant (P < 0.05) after accounting for the volume of mammograms interpreted by each radiologist. CONCLUSIONS: Optimal performance in screening mammography should be evaluated quantitatively. The general pressure to reduce recall rates through "practice guidelines" to below a fixed level for all radiologists should be assessed carefully.

Detection and classification performance levels of mammographic masses under different computer-aided detection cueing environments.

RATIONALE AND OBJECTIVES: The authors evaluated the impact of different computer-aided detection (CAD) cueing conditions on radiologists' performance levels in detecting and classifying masses depicted on mammograms. MATERIALS AND METHODS: In an observer performance study, eight radiologists interpreted 110 subtle cases six times under different display conditions to detect depicted masses and classify them as benign or malignant. Forty-five cases depicted biopsy-proven masses and 65 were negative. One mass-based cueing sensitivity of 80% and two false-positive cueing rates of 1.2 and 0.5 per image were used in this study. In one mode, radiologists first interpreted images without CAD results, followed by the display of cues and reinterpretation. In another mode, radiologists viewed CAD cues as images were presented and then interpreted images. Free-response receiver operating characteristic method was used to analyze and compare detection performance. The receiver operating characteristic method was used to evaluate classification performance. RESULTS: At these performance levels, providing cues after initial interpretation had little effect on the overall performance in detecting masses. However, in the mode with the highest false-positive cueing rate, viewing CAD cues immediately upon display of images significantly reduced average performance for both detection and classification tasks (P < .05). Viewing CAD cues during the initial display consistently resulted in fewer abnormalities being identified in noncued regions. CONCLUSION: CAD systems with low sensitivity (< or = 80% on mass-based detection) and high false-positive rate (> or = 0.5 per image) in a dataset with subtle abnormalities had little effect on radiologists' performance in the detection and classification of mammographic masses.

Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system.

BACKGROUND: Computer-aided mammography is rapidly gaining clinical acceptance, but few data demonstrate its actual benefit in the clinical environment. We assessed changes in mammography recall and cancer detection rates after the introduction of a computer-aided detection system into a clinical radiology practice in an academic setting. METHODS: We used verified practice- and outcome-related databases to compute recall rates and cancer detection rates for 24 Mammography Quality Standards Act-certified academic radiologists in our practice who interpreted 115,571 screening mammograms with (n = 59,139) or without (n = 56,432) the use of a computer-aided detection system. All statistical tests were two-sided. RESULTS: For the entire group of 24 radiologists, recall rates were similar for mammograms interpreted without and with computer-aided detection (11.39% versus 11.40%; percent difference = 0.09, 95% confidence interval [CI] = -11 to 11; P =.96) as were the breast cancer detection rates for mammograms interpreted without and with computer-aided detection (3.49% versus 3.55% per 1000 screening examinations; percent difference = 1.7, 95% CI = -11 to 19; P =.68). For the seven high-volume radiologists (i.e., those who interpreted more than 8000 screening mammograms each over a 3-year period), the recall rates were similar for mammograms interpreted without and with computer-aided detection (11.62% versus 11.05%; percent difference = -4.9, 95% CI = -21 to 4; P =.16), as were the breast cancer detection rates for mammograms interpreted without and with computer-aided detection (3.61% versus 3.49% per 1000 screening examinations; percent difference = -3.2, 95% CI = -15 to 9; P =.54). CONCLUSION: The introduction of computer-aided detection into this practice was not associated with statistically significant changes in recall and breast cancer detection rates, both for the entire group of radiologists and for the subset of radiologists who interpreted high volumes of mammograms.

Optimal reference mammography: a comparison of mammograms obtained 1 and 2 years before the present examination.

OBJECTIVE: We assessed and compared the benefit of using images acquired 1 year or 2 years previously during mammography interpretations. MATERIALS AND METHODS: Eleven radiologists and one resident reviewed 128 cases three times: once without prior mammograms for comparison, once with mammograms from the most recent (1 year) examination, and once with mammograms acquired 2 years previously. They were asked to determine whether the patient should be recalled for additional procedures. Performances under the three conditions were compared. RESULTS: Radiologists were significantly more accurate (p < 0.001) when comparison mammograms (obtained 1 or 2 years previously) were available. Although sensitivity was not significantly affected between the availability of mammograms from 1 or 2 years earlier (p > 0.10), the specificity was. Specificity using mammograms from the latest examination (obtained 1 year previously) as a reference was significantly better (p = 0.03) than specificity using mammograms obtained 2 years previously. CONCLUSION: Comparison mammograms are important for accurate diagnosis-in particular, for increasing specificity. The latest prior examination seems to be the optimal one for this purpose.

Computer-aided detection in mammography: an assessment of performance on current and prior images.

RATIONALE AND OBJECTIVES: The authors assessed and compared the performance of a computer-aided detection (CAD) scheme for the detection of masses and microcalcification clusters on a set of images collected from two consecutive ("current" and "prior") mammographic examinations. MATERIALS AND METHODS: A previously developed CAD scheme was used to assess two consecutive screening mammograms from 200 cases in which the current mammogram showed a mass or cluster of microcalcifications that resulted in breast biopsy. The latest prior examinations had been initially interpreted as negative or definitely benign findings (Breast Imaging Reporting and Data System rating, 1 or 2). The study involved images of 400 examinations acquired in 200 patients. Radiologists identified 172 masses and 128 clusters of microcalcifications on the current images. The performance of the CAD scheme was analyzed and compared for the current and latest prior images. RESULTS: There were significant differences (P < .01) between current and prior images in many feature values. The performance of the CAD scheme was significantly lower for prior than for current images (P < .01). At 0.5 and 0.2 false-positive mass and cluster cues per image, the scheme detected 78 malignant masses (78%) and 63 malignant clusters (80%) on current images. Only 42% of malignant cases were detected on prior images, including 40 masses (40%) and 36 microcalcification clusters (46%). CONCLUSION: CAD schemes can detect a substantial fraction of masses and microcalcification clusters depicted on prior images. To improve performance with prior images, the scheme may have to be adaptively reoptimized with increasingly more subtle abnormalities.